--- title: first snkrfinder.model.cvae keywords: fastai sidebar: home_sidebar nb_path: "nbs/02c_model.cvae.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %}

OVERVIEW: cvae module - convolutional variational auto encoder

TODO:clean up this preamble section preamble: This is a project initiated while an Insight Data Science fellow. It grew out of my interest in making data driven tools in the fashion/retail space I had most recently been working. The original over-scoped idea was to make a shoe desighn tool which could quickly develop some initial sneakers based on choosing some examples, and some text descriptors. Designs are constrained by the "latent space" defined (discovered?) by a database of shoe images. However, given the 3 week sprint allowed for development, I pared the tool down to a simple "aesthetic" recommender for sneakers, using the same idea of utilizing an embedding space defined by the database fo shoe images.

Most of the code is derived from the work of @EtieeneT. (e.g.: TabularData https://github.com/EtienneT/TabularVAE and later https://github.com/EtienneT/vae )

sneaker image encoding via various flavors of auto-encoder.

  • Auto Encoder: AE

    • linear encoder / decoder
    • convolutional encoder / decoder

      • encoders

        • pretrained (e.g. resnet, mobilenet_v2)
        • 'vanilla' convolutional
        • 'vanilla' ResBlock
      • decoders

        • 'vanilla' convolutional
        • 'vanilla' ResBlock
    • Linear Latent representation (LatentLayer)
  • Variational- Auto Encoder: VAE or 𝜷-VAE

    • Variational "bottleneck": "reparameterazation trick" for two variable linear latents
      • 𝜷-VAE: by increasing the relative strenght of error signal of the KLD through a beta variational parameter we can get better formed latent spaces which are somewhat "disentangled". That is have some semantic or meaningful structure in the representation.
    • MMD- variational autoencoder uses a Linear Latent and MMD as the retularizer
      • The KLD has some problems and "over-regularizes" or simply isn't strong enough compared to the Maximum Mean Discrepancy (MMD) loss and tends to converge to a somewhat degenerate solution
      • MD regularization of the latent space has great advantage but at a computational expense

FUTURE EXTENSIONS:

-  model based data cleaning (widget module?)
- GAN finetuning?
- crappify general pattern
- throw out based on inspection of high loss
- try mixed labels?  things that are >50% sneakers included???

Using fastai V2: data pipelining

  • Datablock API

Load the saved merged database, and set the seeds. And doublecheck our data is where we expect.

{% raw %}
{% endraw %} {% raw %}

prep_df_for_datablocks[source]

prep_df_for_datablocks(df)

{% endraw %} {% raw %}
df = prep_df_for_datablocks(df)
{% endraw %} {% raw %}
{% endraw %} {% raw %}

get_ae_btfms[source]

get_ae_btfms(stats='sneaker')

{% endraw %} {% raw %}

get_ae_no_aug[source]

get_ae_no_aug(stats='sneaker')

{% endraw %}

FastAI "block" for autoencoder

Next we need our own version of ReadTabBatch that will return our inputs

{% raw %}
{% endraw %} {% raw %}

class TensorPoint[source]

TensorPoint(x, **kwargs) :: TensorBase

Basic type for points in an image

{% endraw %} {% raw %}

class Tensor2Vect[source]

Tensor2Vect(x, **kwargs) :: TensorPoint

Basic type for points in an image

{% endraw %} {% raw %}

class LatentsTensor[source]

LatentsTensor(x, **kwargs) :: Tensor2Vect

Basic type for latents as Tensor inheriting from TensorPoint (vectors)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

df_get_x[source]

df_get_x(r)

{% endraw %} {% raw %}

df_get_y[source]

df_get_y(r)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

LatentsTensorBlock[source]

LatentsTensorBlock()

{% endraw %} {% raw %}

df_ae_x[source]

df_ae_x(r)

{% endraw %} {% raw %}

df_ae_y[source]

df_ae_y(r)

{% endraw %} {% raw %}

LatentTupleBlock[source]

LatentTupleBlock()

{% endraw %}

Don't forget to set n_inp=1. Otherwise the default to make the input to 1-len(blocks). Also note that the FeatsResize is used to avoid the random jittering from resize during training. Only the very narrow batch augmentations will be used.

{% raw %}
{% endraw %} {% raw %}

get_ae_DataBlock[source]

get_ae_DataBlock(aug=True, im_path=Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data'), stats='sneaker')

wrapper to get the standard ae datablock

{% endraw %}

creating the VAE

Variational Auto-Encoder for fastai

I'm going to use a generic convolutional net as the basis of the encoder, and its reverse as the decoder. This is a proof of concept for using the fastai framework, and will experiment with pre-trained resnet and MobileNet_v2 later. I'd like to use the MobileNet_v2 as a direct link ot the "SneakerFinder" tool which motivated this experiment. [see SneakerFinder]

A variational "module" will sit between the encoder and decoder as the "Bottleneck". The Bottleneck will map the resnet features into a latent space (e.g. ~100 dimensions) represented of standard normal variables. The "reparameterization trick" will sample from this space and the "decoder" will generate images.

Finally a simple "decoder" will sample from the variational latents space and be trained to reconstruct the images.

The intention is the latent space can be used to generate novel sneaker images.

AE: deterministic AutoEncoder ( non- variational)

Although we give up the original utility we are going for -- creating new sneakers via the latent space -- having otherwise equivalent non-variational autoencoders for reference will be great. Furthermore, this latent space representation will be amenable to a MMD regularization later on. This will be useful to avoid some of the limitiations of the KLD as a regularizer (overestimation of variance, and some degenerate convergences). Its sort of hack-ey but keeping the tooling equivalent to the betaVAE will ultimatel give some advantages.

{% raw %}
{% endraw %} {% raw %}

class UpsampleBlock[source]

UpsampleBlock(up_in_c:int, final_div:bool=True, blur:bool=False, **kwargs) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class LatentLayer[source]

LatentLayer(in_features, latent_features) :: Module

This layer encodes the latent "bottleneck" and is constructed to work with the specified VAE DataBlock be a replacement for the variational (reparameter trick) layer for otherwise identical architecture

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class AEEncoder[source]

AEEncoder(arch_body, enc_dim, hidden_dim=None, im_size=160) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %} {% raw %}

class AEDecoder[source]

AEDecoder(hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1]) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %}

It is convenient to avoid the class wrappers to simplify the param splitting for freezing the pre-trained arcitecture.
We could enumerate the class layers and return sequential, but simply making some functions to put the layers togeter is better.

{% raw %}
{% endraw %} {% raw %}

build_AE_encoder[source]

build_AE_encoder(arch_body, enc_dim, hidden_dim=None, im_size=160)

wrapper to sequential-ize AEEncoder class

{% endraw %} {% raw %}

build_AE_decoder[source]

build_AE_decoder(hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1])

wrapper to sequential-ize AEDecoder class

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class AE[source]

AE(enc_parts, hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1]) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %}

Loss

These inherit from Module. (fastai's wrapper on nn.Module). The base class AELoss initializes wiht a batchmean, alpha, and useL1 parameters to set how the loss will be aggregated and regularized. For the basice AutoEncoder we'll regularize the latent with a L1 to keep the magnitudes from exploding.
{% include note.html content='Here batchmean means we will divide the loss (either L1 or L2 depending on useL1 flag) by which uses reduction=’sum’ by the batch size. This technically makes it a cost computed for each batch. This same convention will be used later for KL-Divergence and MMD latent regularizers.' %}

{% raw %}
{% endraw %} {% raw %}

class AELoss[source]

AELoss(batchmean=False, alpha=1.0, useL1=False) :: Module

wrapper for loss_func which deals with potential annealed kl_weight does MSE with 'mean' reduction 'batchmean' averages as 'sum' MSE over batches simple L1 regularizer on latent dimension

{% endraw %}

Metrics

These are called by the recorder callback and collect quantities to track training. I made a simple meta class so I don't have to repreat the reset and mean value property for all the metrics.

{% raw %}
{% endraw %} {% raw %}

class MyMetric[source]

MyMetric() :: Metric

meta-class for simple average over epoch metric quantities

{% endraw %} {% raw %}

class L1LatentReg[source]

L1LatentReg(batchmean=False, alpha=1.0) :: MyMetric

Latent Regularizer with sum reduction and optinal batchmean scaling

{% endraw %}

These are some helpers for computing the KL Divergence as a function and a Module

{% raw %}
{% endraw %} {% raw %}

class KLDiv[source]

KLDiv(batchmean=False) :: Module

Module for computing the KL Divergence from a unit normal distribution. 'batchmean' option sums first and averages over batches

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class L2MeanMetric[source]

L2MeanMetric() :: MyMetric

Mean square error

{% endraw %} {% raw %}

class L1MeanMetric[source]

L1MeanMetric() :: MyMetric

Mean absolute error

{% endraw %} {% raw %}

class L2Metric[source]

L2Metric() :: MyMetric

Sum square error

{% endraw %} {% raw %}

class L1Metric[source]

L1Metric() :: MyMetric

Sum absolute error

{% endraw %} {% raw %}

class L2BMeanMetric[source]

L2BMeanMetric() :: MyMetric

Summed square error average across batch

{% endraw %} {% raw %}

class L1BMeanMetric[source]

L1BMeanMetric() :: MyMetric

Summed abs error average across batch

{% endraw %} {% raw %}

class KLWeightMetric[source]

KLWeightMetric() :: MyMetric

Injected KLD weighting

{% endraw %} {% raw %}

class RawKLDMetric[source]

RawKLDMetric(batchmean=False) :: MyMetric

KLD Metric, batchmean averages across batches

{% endraw %} {% raw %}

class WeightedKLDMetric[source]

WeightedKLDMetric(batchmean=False, alpha=1.0) :: MyMetric

weighted KLD Metric, batchmean averages across batches the "effective" KLD regularization in e.g. a 𝜷-BAE

{% endraw %} {% raw %}

class MuMetric[source]

MuMetric() :: MyMetric

average latent value (e.g. avg(mu)

{% endraw %} {% raw %}

class MuSDMetric[source]

MuSDMetric() :: MyMetric

standard deviation of latent 𝝁 value (e.g. std(mu) )

{% endraw %} {% raw %}

class StdMetric[source]

StdMetric() :: MyMetric

average of latent 𝝈 value (e.g. std(exp(.5*logvar) )

{% endraw %} {% raw %}

class StdSDMetric[source]

StdSDMetric() :: MyMetric

standard deviation of latent 𝝈 value (e.g. std(exp(.5*logvar) )

{% endraw %} {% raw %}

class LogvarMetric[source]

LogvarMetric() :: MyMetric

average of latent log(𝝈*𝝈) value (e.g. mean(logvar))

{% endraw %} {% raw %}

class LogvarSDMetric[source]

LogvarSDMetric() :: MyMetric

standard deviation of latent log(𝝈*𝝈) value (e.g. std(logvar)

{% endraw %}

metrics helpers

{% raw %}
{% endraw %} {% raw %}

default_AE_metrics[source]

default_AE_metrics(alpha, batchmean, useL1)

long-ish default list of metrics for the AE

{% endraw %} {% raw %}

short_AE_metrics[source]

short_AE_metrics(alpha, batchmean, useL1)

short default list of metrics for the AE

{% endraw %}

Callback - AnnealedLoss

The AnnealedLoss Callback basically injects a kl_weight parameter into the loss so we can start training without the full KLD regularization for the beta-VAE version.

The fastai Learner class does the training loop. It took me a little digging into the code to figure out how Metrics are called since its not really stated anywhere in the documentation (Note: create PR for fastai for extra documentation on Metrics logic). By default one of the key Callbacks is the Recorder. It prints out the training summary at each epoch (via ProgressCallBack) and collects all the Metrics. Which by default only loss is a train_met and others are valid_met. The Recorder resets (maps reset() to all mets) the metrics before_train and before_valid. The Recorder maps accumulate() to the metrics on after_batch. Finally

AnnealedLossCallback will inject the latent mu and logvar and a kl_weight variable into our loss. The mu and logvar will be used to compute the KLD. The kl_weight is a scheduled weighting for the KLD. You can see the schedule graph of the parameter. At the beginning it will be 0, thus the KLD part of the loss will get ignored. So during 10% of training, we will fit a normal auto-encoder. Then gradually for 30% of trainning, increase kl_weight to 1 and then remain there for the remaining training time so that the auto encoder now becomes full variational. The way this callback is done, the loss will receive this parameter, but not the model.

{% raw %}
{% endraw %} {% raw %}

class AnnealedLossCallback[source]

AnnealedLossCallback(after_create=None, before_fit=None, before_epoch=None, before_train=None, before_batch=None, after_pred=None, after_loss=None, before_backward=None, before_step=None, after_cancel_step=None, after_step=None, after_cancel_batch=None, after_batch=None, after_cancel_train=None, after_train=None, before_validate=None, after_cancel_validate=None, after_validate=None, after_cancel_epoch=None, after_epoch=None, after_cancel_fit=None, after_fit=None) :: Callback

injects kl_weight for access during loss function calculation

{% endraw %} {% raw %}

default_KL_anneal_in[source]

default_KL_anneal_in()

reasonable default for 'warming up' the KL Div

{% endraw %} {% raw %}
n_epochs = 10
f_init = combine_scheds([.1, .7, .2], [SchedNo(0,0),SchedCos(0,1), SchedNo(1,1)])
# f = combine_scheds([.8, .2], [SchedCos(0,0), SchedCos(0,.5)])
p = torch.linspace(0.,1.,100)
pp = torch.linspace(0.,1.*n_epochs,100)

plt.plot(pp,[f_init(o) for o in p])
[<matplotlib.lines.Line2D at 0x7f4e01251a00>]
{% endraw %}

{% include warning.html content='Avoid using early stopping because the AnnealedLossCallback will make the loss grow once the KL divergence weight kicks in. ' %}

I want to note something here that was a little confusing to me: params(model) is a builtin fastai PyTorch.core function which returns all of the parameters of the modules. i.e.

def params(m):
    "Return all parameters of `m`"
    return [p for p in m.parameters()]

The toplevel fastai core functions with simple names that almost match class attributes was one of my biggest stumbling blocks in getting acquainted with the fastai v2 API. (The other is the documentation which is autogenerated by the fastdev frameworks from their development noteboooks. More on that struggle and my tips if that is troblesome for you later (here). {% include note.html content='that it is crucial that you don’t freeze the batch norm layers. The bn_splitter collects out all the batchnorm layers. The simple splitting we do only freezes the encoder and leaves the latent layers (i.e. VAE or linear encoding bottlenedck) and the decoder in a parameter group with the batchnorm layers.' %}

parameter Splitters

{% include warning.html content='there are two completely different splitters in the FastAI API. This splitter groups the model parameters into groups for freezing and for progressive learning rates. (The other one is splits data into train and validate. I got imminiently confused when I first started with the API by this.' %}

TODO:more sophisticated parameter splitting to enable progressive learning rates

{% raw %}
{% endraw %} {% raw %}

bn_splitter[source]

bn_splitter(m)

splits all the batchnorm layers out

{% endraw %} {% raw %}

resnetVAE_split[source]

resnetVAE_split(m)

simple splitter to freeze the non batch norm pre-trained encoder

{% endraw %} {% raw %}

AE_split[source]

AE_split(m)

generic splitter for my AE classes- BVAE & AE & MMDVAE.

{% endraw %}

other encoder and decoder types

  1. MobileNet_v2 as the encoder, as a continuation of the original Sneaker Finder

  2. simple bowtie convolutional encoder / decoder (Mimics the GOAT medium blog)

    • Architecture Hyperparameters:- Latent Size (research default 256, production default 32) - Filter Factor Size (research default 16, production default 32)
      • Latent Linear Hidden Layer Size (research default 2048, production default 1024)
      • The encoder architecture is as follows with research defaults from above:
        • Input 3x128x128 (conv2d block [conv2d, batchnorm2d, relu])
        • 16x64x64 (conv2d block [conv2d, batchnorm2d, relu])
        • 32x32x32 (conv2d block [conv2d, batchnorm2d, relu])
        • 64x16x16 (conv2d block [conv2d, batchnorm2d, relu])
        • 128x8x8 (conv2d block [conv2d, batchnorm2d, relu])
        • Flatten to 8192
        • 2048 (linear block [linear, batchnorm1d, relu])
        • Split the 2048 dimension into mu and log variance for the parameters of the latent distribution
        • Latent mu size 256 (linear layer only with bias)
        • Latent logvar size 256 (linear layer only with bias)
      • In the middle here you can break out the BCE and KLD loss for the final loss term and use the standard reparam trick to sample from the latent distribution.
      • Decoder architecture an exact mirror
        • Input 256
        • 2048 (linear block [linear, relu])
        • 8192 (linear block [linear, batchnorm1d, relu])
        • reshape (128x8x8)
        • 64x16x16 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])
        • 32x32x32 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])
        • 16x64x64 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])
        • 3x128x128 (conv2d transpose [convtranspose2d, sigmoid]
      • For weight initialization I used a normal distribution centered at zero with 0.02 set for the stddev. Optimizer: Adam with default parameters, if I were to do it over again I'd spend more time here understanding the learning dynamics. The dataset was about ~10,000 with a 70/20/10 split, batch size 64, over 120 epochs, with a learning schedule to reduce when the loss started to plateau. No crazy image augmentation just resizing and standards flips. I used the ANN package Annoy to do the NN search for prod, normalizing the embeddings and using the cosine similarity, ANN factor was 128 for num_trees.
  1. MMD regularized VAE where the latents are drawn from a

TODO:Ranger optimizer might really help .. test

We can also use the transfer learning VAE tooling we previously built. We just need to create the convolutional encoder and pass it in... Note that we don't have a pre-trained option, so DON'T FREEZE!

Now just wrap that simple conv block architecture into a builder. And a meta-wrapper to let us call the conv_encoder and pre-trained options with the same function. (I'll also put the get_pretrained_parts function here now even though we won't use it till the next section, so that we can make the get_encoder_parts generic wrapper handle both properly.)

{% raw %}
{% endraw %} {% raw %}

get_conv_parts[source]

get_conv_parts(im_size=160)

make a simple convolutional ladder encoder

{% endraw %} {% raw %}

get_pretrained_parts[source]

get_pretrained_parts(arch='resnet18')

this works for mobilnet_v2, resnet, and xresnet

{% endraw %} {% raw %}

get_encoder_parts[source]

get_encoder_parts(enc_type='vanilla')

{% endraw %} {% raw %}
##   


latent_dim = 128

# equalize KLDiv wrt errors per pixel
alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha /= 20  # 5% retularizer

batchmean = True 
useL1 = False
hidden_dim = None
# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),  ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

metrics = default_AE_metrics(alpha,batchmean,useL1)

block = get_ae_DataBlock(aug=True)
batch_size = 64
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'

vae = AE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = AELoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %}

{% include note.html content='The to_fp16() callbacks work but increasing the batch size doesn’t really speed things up.' %}

{% raw %}
latent_dim = 128

# equalize KLDiv wrt errors per pixel
alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha /= 20  # 5% retularizer

batchmean = True 
useL1 = False
hidden_dim = None
# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),  ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

metrics = default_AE_metrics(alpha,batchmean,useL1)

block = get_ae_DataBlock(aug=True)
batch_size = 64
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'

vae = AE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = AELoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
learn = learn.to_fp16()
{% endraw %}

constructing VAE with Module Class Layers

For several of the decoder and "sampler" layers I might want to turn off the nonlinearity to give us more reasonable "gaussian" outputs to the Variational layer and the generated image which will is compared with the ImageNetStats batch-normalized image.

IMPORTANT VAE TIP!!! Make sure NOT to use batch normalization and non-linearity in the linear layers of the VAE. The normalization will affect the representation and the KLD constraints.

{% raw %}
{% endraw %} {% raw %}

class VAELinear[source]

VAELinear(in_features, latent_features) :: Module

maps hidden (input) features to two latents (mu and logvar)

{% endraw %} {% raw %}

class VAELayer[source]

VAELayer(in_features, latent_features) :: Module

The VAE : in_features to latent_features through the "Variational" magic: "reparamaterization trick"

{% endraw %}

simple BVAE class from VAE layer class components

This creates a pair of latents from which we can perform the "resample" trick. {% include note.html content='this is a $\beta$-VAE (hence BVAE because we have a weighting factor for the KL Divergence regularazation factor which acts as a Legrangian).' %}

Putting it all together gives us our VAE! Note that we'll pass in the "parts" of the encoder for ease of using pretrained (or not) architectures. The model name will correspond to the architecture of the encoder via name.

Note that the BVAE can simply inherit from the AE class we defined above. Really the only difference in the __init__ function is that a VAELayer which performs the reparameterization trick replaces the AElayer as self.bn

{% raw %}
{% endraw %} {% raw %}

class BVAE[source]

BVAE(enc_parts, hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1]) :: AE

simple VAE made with an encoder passed in, and some builder function for the Latent (VAE reparam trick) and decoder

{% endraw %}

A nice wrapper for building the encoder parts will be handy.

{% raw %}
{% endraw %} {% raw %}

get_pretrained_parts[source]

get_pretrained_parts(arch='resnet18')

this works for mobilnet_v2, resnet, and xresnet

{% endraw %}

Sweet, we've verified the arcitecture works, but we need to train it with a loss that constrains the variational layers with the KL Divergence. Otherwise the simple MSE will diverge.

VAE Loss functions classes

This simply adds a KLD regularizer to the latent space defined by two rank-1 tensors defining gaussian-prior latents. i.e. a mean ($\mu$) and standard deviation ($\sigma$).
{% include note.html content='for convenience and numeric stability the $\sigma$ is representated as a $\log(\sigma^{s})$ so the tensors are called mu and logvar' %}

{% raw %}
{% endraw %} {% raw %}

class BVAELoss[source]

BVAELoss(batchmean=False, alpha=1.0, useL1=False) :: Module

Measures how well we have created the original image, plus the KL Divergence with the unit normal distribution batchmean option sums first and averages over batches (for smaller total error magnitudes.. cosmentic)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

default_VAE_metrics[source]

default_VAE_metrics(alpha, batchmean, useL1)

long default list of metrics for the VAE

{% endraw %} {% raw %}

short_VAE_metrics[source]

short_VAE_metrics(alpha, batchmean, useL1)

short default list of metrics for the AE

{% endraw %}

Here's how we put everything together.

{% raw %}
latent_dim = 128

alpha = 5
batchmean = True
useL1 = False
hidden_dim = None


cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]

metrics = default_VAE_metrics(alpha,batchmean,useL1)
{% endraw %}

We can use this callback if we want to save the model at every epoch. Which could be super useful if we were able to actually overfit our model.

'SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True)'

{% raw %}
# NOTE: lf_finder does NOT work correctly with our annealed kl_weight... 

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.0002511886414140463,
 0.0003311311302240938,
 0.00029115988581907004,
 0.00028840304003097117)
{% endraw %}

{% include note.html content='The CallBacks need to be updated for the KL loss annealing schedule as we tune. We want to turn the kl_weight to 1.0 for instance when using the learning rate finder learn.lr_find(), and after an initial burn in where the KL_loss term gradually ramps in, setting the kl_weight to stay at unity will be useful as we separately turn the learning rate (e.g. fit_one_cycle or fit_flat_cosine.' %}

{% raw %}
#put in the annealied KL_weight...
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler

learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )
<fastai.learner.Learner at 0x7f4e019d9e20>
{% endraw %} {% raw %}
n_epochs = 5

#learn.fit_one_cycle(n_epochs,lr_max= lr1)
learn.fit_one_cycle(n_epochs)
learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 61706.023438 59401.890625 59200.527344 -0.183162 0.670351 -1.360018 1074.730103 0.182537 64435.050781 1.328313 0.744954 1.561802 00:22
1 58114.496094 54704.398438 54106.003906 -0.067606 0.709372 -1.241548 981.183960 0.604076 61621.859375 1.449838 0.450837 1.893286 00:22
2 55027.734375 52449.945312 52048.351562 -0.016363 0.778369 -0.767656 417.512909 0.947244 60466.074219 0.887459 0.376451 1.181295 00:22
3 53234.660156 51567.859375 51215.320312 -0.014475 0.771869 -0.710858 346.533447 1.000000 59968.601562 0.823350 0.310374 1.014969 00:22
4 52392.648438 51421.828125 51092.546875 -0.048797 0.782629 -0.654477 323.230713 1.000000 59896.394531 0.809924 0.289300 0.944704 00:22
{% endraw %}

The vae with pretrained resnet encoder seems to train to a much better end-point if we keep the resnet frozen. Hence the commented out learn.unfreeze() below.

{% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()
<fastai.learner.Learner at 0x7f4e019d9e20>
{% endraw %} {% raw %}
epochs = 20

#learn.fit_one_cycle(epochs, lr_max= 1e-3)
learn.fit_flat_cos(epochs,pct_start=.05)
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 73116.570312 56011.906250 41707.714844 -0.162279 0.150928 -43.849857 14298.664062 1.000000 53618.125000 1.220086 0.524610 36.704914 00:22
1 55711.816406 45964.726562 33920.769531 -0.043475 0.279428 -35.963737 12039.008789 1.000000 47239.265625 1.312494 0.757988 37.766373 00:22
2 44836.546875 37305.648438 27406.685547 -0.023731 0.321513 -29.381834 9895.095703 1.000000 41459.550781 1.460417 0.538946 35.474781 00:22
3 36200.839844 30034.455078 21493.714844 -0.048967 0.376191 -24.223383 8544.314453 1.000000 35556.191406 1.591922 0.687644 33.419842 00:22
4 29243.531250 26322.974609 17289.226562 -0.076567 0.462568 -20.003771 9098.589844 1.000000 30504.798828 1.651102 1.249426 31.537554 00:23
5 23948.765625 20693.746094 14159.992188 0.015880 0.476163 -17.972431 6532.034668 1.000000 27060.267578 1.679238 0.613610 30.386091 00:22
6 18914.341797 16118.884766 10100.545898 0.071780 0.485414 -15.699727 6016.059082 1.000000 21769.064453 1.614907 0.813107 28.720329 00:22
7 16302.161133 16133.471680 9065.597656 -0.032543 0.528036 -14.508196 7064.763672 1.000000 20103.945312 1.672818 0.956630 27.728195 00:23
8 14035.663086 13486.658203 8366.876953 -0.014977 0.549360 -13.761081 5118.403809 1.000000 18096.849609 1.596557 0.572698 27.181425 00:23
9 12555.615234 12438.543945 7636.028809 -0.001427 0.551520 -12.375941 4801.065430 1.000000 16884.033203 1.493703 0.722778 25.457499 00:22
10 11842.099609 11512.832031 7183.715820 0.014809 0.537542 -11.792240 4328.155762 1.000000 15679.622070 1.466000 0.521400 24.649998 00:22
11 10849.121094 10924.213867 6975.569824 -0.034562 0.546702 -10.934856 3947.825195 1.000000 15087.824219 1.373056 0.457514 23.260689 00:22
12 10149.152344 10338.746094 6664.760742 0.058513 0.561050 -10.111523 3673.358154 1.000000 14550.804688 1.332573 0.493117 22.023117 00:22
13 9612.157227 9841.376953 6494.279785 -0.028188 0.523342 -9.359609 3346.479980 1.000000 14008.334961 1.275519 0.430882 20.670998 00:22
14 9164.250977 9458.142578 6349.508789 0.017441 0.548174 -8.786793 3108.029785 1.000000 13718.131836 1.209039 0.398604 19.754648 00:22
15 8761.227539 9177.581055 6264.538574 0.044239 0.542927 -8.290083 2912.659668 1.000000 13411.566406 1.172588 0.369216 18.736010 00:22
16 8444.499023 8896.747070 6152.080078 -0.001141 0.557176 -7.828480 2744.129395 1.000000 13307.772461 1.127717 0.398963 17.916582 00:22
17 8178.131836 8776.557617 6145.216309 0.005876 0.548124 -7.555161 2630.786133 1.000000 13287.386719 1.102357 0.376227 17.280994 00:22
18 8179.623535 8713.017578 6129.683105 0.007566 0.553797 -7.419802 2582.790527 1.000000 13186.386719 1.086193 0.387691 17.034954 00:22
19 7979.851562 8669.065430 6088.716309 0.003607 0.549836 -7.418787 2579.850342 1.000000 13166.111328 1.084882 0.384289 17.028463 00:23
{% endraw %} {% raw %}
learn.show_results()
{% endraw %}

"Vanilla" convolutional beta-VAE

{% raw %}
latent_dim = 128

# equalize KLDiv wrt errors per pixel
#  alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha = 5
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight': default_KL_anneal_in() })]

metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'
vae = BVAE(get_encoder_parts(arch), hidden_dim=None,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %}

MMD-VAE

The MMD replace latent regularization term in loss_fn (KLD) with Maximal Meanm Discrepancy.

We'll make an MMDVAE class to keep things declarative, but its really just an AE. i.e. a linear latent layer.

Additional background on MME from [https://github.com/Saswatm123/MMD-VAE]:

Maximum Mean Discrepancy Variational Autoencoder, a member of the InfoVAE family that maximizes Mutual Information between the Isotropic Gaussian Prior (as the latent space) and the Data Distribution.\ Short explanation:The traditional VAE is known as the ELBO-VAE, named after the Evidence Lower Bound used in its objective. The ELBO suffers from two problems: overestimation of latent variance, and uninformative latent information.\The latter is because one of the objective's terms is the KL-Divergence between the Gaussian parameterized by the encoder and the Standard Isotropic Gaussian. This dissuades usage of the latent code, so that the KL-Divergence term is allowed to fall even further. It is important to note that the KL-Divergence should never truly reach zero, as that means the encoder is not learning useful features and cannot find feature locality, and the decoder is just randomly sampling from Standard Gaussian noise.\ The overestimation of variance results from the KL-Divergence term not being strong enough to balance against the Reconstruction Error, and thus the Encoder prefers to learn a multimodal latent distribution with spread apart means, leading to low training error as it overfits, but low quality samples as well, as the sampling distribution is assumed to be a Standard Isotropic Gaussian. One effort to mitigate this effect is the Disentangled Variational Autoencoder, which simply raises the weight on the KL-Divergence term. However, this increases the problem stated in the paragraph above since it further penalizes using the latent code.\ For more detailed explanations, I used these resources to learn, in order of usefulness to me:\

- https://arxiv.org/pdf/1706.02262.pdf \
- http://ruishu.io/2018/03/14/vae/ \
- http://approximateinference.org/accepted/HoffmanJohnson2016.pdf \
- https://ermongroup.github.io/blog/a-tutorial-on-mmd-variational-autoencoders/ \
- http://bjlkeng.github.io/posts/variational-bayes-and-the-mean-field-approximation/ \
- https://ermongroup.github.io/cs228-notes/inference/variational/ \
{% raw %}
{% endraw %} {% raw %}

class MMDVAE[source]

MMDVAE(enc_parts, hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1]) :: AE

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %} {% raw %}

gaussian_kernel[source]

gaussian_kernel(a, b)

{% endraw %} {% raw %}

class MaxMeanDiscrepancy[source]

MaxMeanDiscrepancy(batchmean=False) :: Module

MMD add alpha?

{% endraw %} {% raw %}

class MMDLoss[source]

MMDLoss(batchmean=False, alpha=1.0, useL1=False) :: Module

Measures mean square error of prediction and original image, regularized by MMD.

Note: using reuction = 'mean' because it keeps the regularization relatively potent (i.e. pixels>>latents)

{% endraw %} {% raw %}

class MMDMetric[source]

MMDMetric(batchmean=False, alpha=1.0) :: MyMetric

meta-class for simple average over epoch metric quantities

{% endraw %} {% raw %}

short_MMEVAE_metrics[source]

short_MMEVAE_metrics(alpha, batchmean, useL1)

short list of metrics for the VAE

{% endraw %} {% raw %}

default_MMEVAE_metrics[source]

default_MMEVAE_metrics(alpha, batchmean, useL1)

long default list of metrics for the VAE

{% endraw %}

Vanilla MMD VAE

Simply call our get_encoder_parts with arch='vanilla' in the MMDVAE builder.

{% raw %}
latent_dim = 128

cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
alpha = 20

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
{% endraw %}

ResNet Encoder MMDVAE

We build these by passing resnet18 (not 'resnet18') to get_encoder_parts helper for the parts to init the MMDVAE.

First a traditional "fine_tune" type of training:

{% raw %}
latent_dim = 128

cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
learn.freeze()
{% endraw %} {% raw %}
learn.freeze()
n_epoch = 20
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(n_epoch)#,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()
{% endraw %} {% raw %}
n_epoch = 20
learn.unfreeze()
#learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.7)
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()
{% endraw %}

vanilla-resnet MMD VAE

no freeze

We build these by passing resnet18 (not 'resnet18') to get_encoder_parts helper for the parts to init the MMDVAE.

{% include note.html content='our architecture trains best when simply starting with the pretrained weights. Trying to "fine_tune" by training the backend on a frozen resnet and then unfreezing doesn’t work with the parameter groupings from the AE_split. The learn.unfreeze() doesn’t actually do anythign (unfrozen is default) but makes is clear we are un -frozen' %}

{% raw %}
latent_dim = 128

cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]


alpha = 10

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
    
    
learn.unfreeze()
{% endraw %}

vanilla-resnet MMD-VAE

ResBlocks replacing Conv2d in encoder AND decoder.

'Vanilla' encoder architecture made with ResBlock layers instead of ConvLayer layers and Mish activation.

{% raw %}
{% endraw %} {% raw %}

class UpsampleResBlock[source]

UpsampleResBlock(up_in_c:int, final_div:bool=True, blur:bool=False, **kwargs) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %} {% raw %}

get_resblockencoder_parts[source]

get_resblockencoder_parts(enc_type='vanilla', im_size=160)

make a simple (hence 'vanilla') convolutional ladder encoder with ResBlock parts

{% endraw %} {% raw %}

class ResBlockAEDecoder[source]

ResBlockAEDecoder(hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1]) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %} {% raw %}

build_ResBlockAE_decoder[source]

build_ResBlockAE_decoder(hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1])

wrapper to sequential-ize AEDecoder class

{% endraw %} {% raw %}

class ResBlockAE[source]

ResBlockAE(enc_parts, hidden_dim=None, latent_dim=128, im_size=160, out_range=[-1, 1], isVAE=False) :: AE

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %}

Finally we can use the ResBlockBVAE which instantiates a ResBlock decoder to optimize the architecture. This is following the fastAI API lessong from the "bag of tricks" ResNet paper (CITATION), which is a general true-ism which could be glibly states as: "replacing a Conv with a ResBlock always gets you better results". The Class constructor ResBlockAE takes isVAE to switch between AELayer and LatentLayer latents.

TODO:update AE class to make a VAE or AE based on the isVAE switch

{% raw %}
latent_dim = 128
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE,isVAE= True)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %}

next steps.

  • visualize activations in encoder decoder
  • improve training with progressive learning rates
  • G

export / provenance

128 latents, alpha=10

{% raw %}
latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
     
    
    
    

    
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.33113112449646,
 0.03981071710586548,
 0.18547092080116273,
 0.11481534689664841)
{% endraw %} {% raw %}
#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mmd mu std logvar l1_b_mean mu_sd l1_latent_reg weighted_kld logvar_sd time
0 35639.734375 33119.378906 32758.919922 354.753448 -0.645665 2.168537 1.389641 45340.976562 5.047215 5219.156250 18799.441406 0.807266 00:24
1 22419.580078 15466.881836 15300.407227 162.598511 -0.390403 2.085497 1.324767 30500.722656 3.677496 3760.603516 10589.691406 0.736531 00:23
2 14881.570312 10914.019531 10817.428711 94.298996 -0.218614 1.718532 0.952760 24908.886719 2.857116 2908.460693 6201.050781 0.693021 00:23
3 10775.079102 8509.122070 8455.727539 51.379539 -0.116926 1.412444 0.578432 21353.609375 2.194111 2228.759521 3532.319580 0.640258 00:24
4 8417.246094 7147.596680 7119.229980 26.634216 -0.072432 1.150160 0.179836 18936.121094 1.669925 1692.649536 1983.132935 0.602620 00:24
5 6787.373047 6221.455566 6203.451660 16.419044 -0.051861 0.946574 -0.200824 17108.943359 1.339317 1349.057983 1275.688110 0.569902 00:23
6 5708.291504 5401.690430 5387.585449 12.589787 -0.049178 0.882773 -0.334380 15516.946289 1.213070 1216.506592 1068.951294 0.555498 00:23
7 4967.996582 4810.905273 4799.205566 10.367170 -0.035553 0.831857 -0.451990 14039.386719 1.123102 1120.465332 946.840881 0.552006 00:24
8 4480.203613 4512.324219 4501.353516 9.820558 -0.032393 0.812645 -0.496762 13347.663086 1.094568 1089.507446 911.750488 0.546974 00:23
9 4122.484863 4195.836426 4185.203613 9.572933 -0.030994 0.816746 -0.486494 12404.958984 1.091210 1083.963379 904.418335 0.547605 00:24
10 3843.719238 3955.936768 3945.632324 9.421382 -0.027061 0.806554 -0.510669 11746.567383 1.079526 1070.157715 891.319214 0.544316 00:24
11 3627.255371 3750.461670 3739.786133 9.974444 -0.035787 0.810802 -0.502045 11090.406250 1.095940 1082.943848 915.384888 0.550083 00:24
12 3453.144775 3555.236572 3544.947998 9.497383 -0.033756 0.805314 -0.512734 10532.422852 1.081325 1065.357910 893.720581 0.540843 00:24
13 3317.225586 3445.394287 3435.099854 9.603128 -0.026698 0.807787 -0.506707 10088.870117 1.086125 1070.625977 898.874268 0.540827 00:23
14 3207.483887 3477.685303 3467.043945 9.804630 -0.036560 0.803424 -0.516729 10080.664062 1.088416 1068.231079 904.268982 0.537274 00:23
15 3097.683838 3296.834473 3286.019287 10.156882 -0.024965 0.809974 -0.501416 9402.141602 1.100944 1077.333496 919.126099 0.541100 00:24
16 3026.498535 3239.862305 3229.419434 9.751334 -0.027817 0.814940 -0.486714 9327.415039 1.095807 1070.757568 907.116394 0.533075 00:23
17 2942.689209 3145.737793 3134.896729 10.162513 -0.031712 0.819668 -0.476477 8857.916992 1.109087 1079.275024 925.773315 0.536413 00:24
18 2910.235596 3092.333984 3081.208252 10.555488 -0.016682 0.829229 -0.454906 8585.517578 1.126567 1096.386475 948.013977 0.542863 00:23
19 2832.958252 3017.066650 3006.289307 10.245089 -0.018359 0.826162 -0.462675 8570.823242 1.114929 1082.901733 933.252991 0.543466 00:23
20 2772.829590 3046.259277 3036.564209 9.047776 -0.024695 0.789895 -0.547899 8529.547852 1.059283 1029.493530 866.595886 0.529360 00:24
21 2753.147461 2986.807861 2975.513428 10.669466 -0.013018 0.844861 -0.416567 8073.350098 1.141360 1104.890015 962.535461 0.538116 00:23
22 2712.455566 2906.891602 2896.313965 10.020215 -0.016617 0.819964 -0.475952 8024.366211 1.107157 1071.144531 922.119934 0.536537 00:23
23 2677.711914 3020.316895 3009.638916 10.101330 -0.021751 0.810849 -0.496884 8298.047852 1.099663 1062.236694 914.141846 0.532246 00:23
24 2626.546875 2862.383057 2852.536377 9.309927 -0.015195 0.803757 -0.513902 8120.997070 1.077071 1039.144897 884.831543 0.530815 00:23
25 2575.743164 2870.408691 2860.929199 9.051567 -0.026579 0.797159 -0.528016 8179.134766 1.062291 1025.824219 865.369934 0.521008 00:23
26 2480.682617 2658.409668 2648.303711 9.851988 -0.019407 0.812116 -0.495792 7458.644043 1.096348 1055.604858 911.445984 0.536942 00:24
27 2367.523193 2564.386475 2555.031738 9.078519 -0.010155 0.785778 -0.558931 7359.857910 1.055575 1014.959229 864.135681 0.527093 00:23
28 2287.362793 2577.752686 2568.370605 8.995213 -0.011609 0.781813 -0.570015 7215.798340 1.051848 1011.241882 862.347351 0.529978 00:24
29 2229.250488 2449.017822 2440.301514 8.443208 -0.008403 0.766377 -0.607935 7179.539551 1.025925 984.454834 834.755188 0.523161 00:23
30 2194.789062 2457.297119 2448.329590 8.684980 -0.008512 0.759338 -0.627476 7038.735840 1.025120 981.841309 838.818848 0.527536 00:24
31 2152.750732 2365.702148 2356.650146 8.731876 -0.015857 0.768805 -0.602370 6879.909668 1.036128 993.249329 847.809448 0.525540 00:23
32 2104.733643 2339.498535 2330.842041 8.281164 -0.006147 0.755411 -0.635174 6761.827637 1.013497 971.661621 823.330444 0.517066 00:24
33 2079.943604 2352.695068 2343.459961 8.989347 0.000814 0.767590 -0.606251 6705.492676 1.039513 994.690674 854.078735 0.525999 00:23
34 2057.652100 2289.373535 2281.190186 8.049372 -0.003593 0.754654 -0.636665 6767.381348 1.005222 961.257935 812.971985 0.514499 00:23
35 2048.321533 2333.577881 2325.027100 8.287884 -0.004861 0.745774 -0.659917 6811.984375 1.004853 959.555969 817.796509 0.513304 00:23
36 2008.904785 2287.277344 2278.947998 8.144099 -0.007205 0.744801 -0.662207 6504.483887 1.000480 952.984863 812.662354 0.512287 00:24
37 1978.226929 2199.820557 2191.496582 8.082468 -0.014684 0.740286 -0.675295 6599.623535 0.994912 947.844360 809.869019 0.514765 00:24
38 1965.509399 2354.071045 2345.678467 8.060910 -0.011115 0.751032 -0.645638 6712.160156 1.002705 954.138611 811.483521 0.512083 00:23
39 1944.204224 2543.402832 2531.188232 11.698755 -0.002674 0.802894 -0.519100 6909.925781 1.131840 1088.012329 967.922241 0.531437 00:23
40 1927.125122 2166.637451 2157.772949 8.694813 0.004934 0.754433 -0.638231 6360.273926 1.020341 970.247681 834.338135 0.515969 00:24
41 1904.941650 2148.423584 2138.998535 9.175574 -0.004919 0.763749 -0.616479 6247.569824 1.041988 990.484131 860.605591 0.524700 00:23
42 1880.683228 2143.890137 2136.084473 7.667233 -0.005195 0.724807 -0.717712 6384.495117 0.971001 920.460571 791.137024 0.514054 00:23
43 1868.364868 2138.596680 2130.624268 7.770609 -0.003167 0.732386 -0.696039 6208.037109 0.981391 929.703735 797.666382 0.510722 00:23
44 1873.806519 2115.231445 2106.587646 8.408823 0.002722 0.759259 -0.626507 6110.434082 1.019319 965.254517 830.953125 0.517964 00:23
45 1857.733521 2118.644287 2109.715576 8.607017 -0.003585 0.758680 -0.628329 6144.351074 1.024203 970.119385 838.559998 0.518992 00:24
46 1835.785889 2301.718750 2293.943604 7.280443 0.002950 0.731905 -0.693791 6752.790039 0.966830 915.910034 775.821228 0.498173 00:24
47 1826.181519 2070.978271 2062.959473 7.751598 -0.006774 0.736675 -0.683394 6289.015137 0.984957 930.847839 798.094299 0.506730 00:24
48 1816.817261 2169.010010 2158.630859 10.077996 0.000032 0.782318 -0.571635 6146.196777 1.079840 1023.483887 905.510681 0.533880 00:23
49 1790.655762 2093.010254 2083.828613 8.849428 -0.004012 0.751683 -0.650315 6050.121094 1.025513 967.276062 848.441895 0.530520 00:23
50 1793.118286 2089.919434 2081.775391 7.915750 -0.009043 0.746823 -0.658815 6050.777344 0.996809 939.305908 809.562744 0.515972 00:24
51 1784.936768 1993.558228 1985.290527 8.115150 0.003885 0.743259 -0.670213 5850.540527 0.999537 940.279785 817.424438 0.521515 00:23
52 1767.381958 2151.812012 2144.149658 7.295479 -0.002681 0.714391 -0.744198 6923.266113 0.954031 902.872864 775.648926 0.504909 00:24
53 1756.909668 2133.470947 2125.688232 7.532532 0.006581 0.730068 -0.702234 6150.624512 0.973271 915.361450 789.177856 0.508780 00:23
54 1746.442627 2092.561768 2084.928467 7.357768 -0.000443 0.728496 -0.706767 6296.643555 0.968090 909.576660 784.124512 0.510359 00:24
55 1744.525879 2060.050049 2051.518799 8.250493 -0.002866 0.753495 -0.642588 5905.169922 1.010369 948.708313 824.126892 0.520684 00:24
56 1725.987915 2222.684082 2211.968994 10.327498 0.001512 0.780892 -0.574243 6259.079590 1.084146 1023.592651 910.862427 0.530152 00:24
57 1713.938354 2353.004395 2345.289307 7.142781 -0.002146 0.695641 -0.793052 8155.063477 0.933386 889.344971 761.682495 0.489615 00:23
58 1705.436157 1990.490601 1982.330688 7.960275 0.006327 0.741737 -0.672134 5864.037109 0.992518 932.534546 807.071838 0.514355 00:24
59 1705.450562 2019.981323 2010.744507 8.982873 -0.000409 0.759453 -0.628868 5937.989746 1.035373 972.338074 855.795715 0.526218 00:24
60 1696.698486 1959.749512 1951.297974 8.261941 -0.000039 0.743128 -0.669158 5735.625488 1.001879 939.263184 818.803406 0.516575 00:24
61 1696.808472 2162.159180 2151.416260 10.234275 0.011562 0.773763 -0.594086 6051.969727 1.078619 1015.008423 909.483826 0.533412 00:24
62 1681.989136 1939.605103 1931.698853 7.813064 -0.000969 0.728840 -0.709018 5738.298340 0.978495 914.541321 800.384216 0.520435 00:24
63 1667.958008 1908.849731 1901.618042 6.975263 0.005093 0.717563 -0.736385 5811.500488 0.948977 888.578186 768.285400 0.507422 00:24
64 1657.080566 1997.998535 1990.892212 6.848620 0.003718 0.708017 -0.761569 6328.044922 0.936035 878.318420 758.598022 0.501764 00:23
65 1650.266357 1880.462524 1872.814941 7.524014 -0.000007 0.728828 -0.709459 5613.864746 0.972260 908.505615 793.400818 0.520965 00:23
66 1644.701172 1923.961914 1916.933350 6.810940 0.001214 0.712433 -0.750617 5899.688477 0.939393 879.358093 760.829346 0.506575 00:24
67 1660.577271 1908.370361 1901.273438 6.838502 -0.000657 0.715031 -0.742947 5754.287109 0.942613 881.583496 762.272522 0.505253 00:23
68 1642.703491 2075.767090 2068.711670 6.701159 0.004295 0.686344 -0.822066 6988.598633 0.914129 860.682068 749.578125 0.495988 00:23
69 1637.802124 1936.513306 1927.819946 8.439760 0.001987 0.747285 -0.662898 5694.429199 1.013492 942.569397 836.994812 0.531298 00:24
70 1628.861938 1855.664185 1848.401855 7.107107 0.004089 0.721452 -0.729289 5562.158203 0.956804 892.012207 779.056335 0.518865 00:23
71 1620.777588 1849.363525 1841.987549 7.230100 0.005408 0.727281 -0.712910 5576.687988 0.964420 897.751221 783.769714 0.518859 00:23
72 1617.906738 1872.577148 1865.401001 7.024238 0.001375 0.722295 -0.724200 5726.461914 0.953735 888.874390 771.711609 0.510274 00:23
73 1605.725342 1879.347168 1872.517212 6.652784 0.001630 0.705851 -0.771451 5707.089355 0.931615 866.731323 759.250916 0.513576 00:24
74 1601.359619 1860.889526 1853.274292 7.449453 0.000522 0.732364 -0.699802 5518.531738 0.974449 906.497131 793.526978 0.520268 00:23
75 1593.129028 1819.535645 1812.225342 7.186706 -0.003417 0.723173 -0.724752 5449.743164 0.959148 890.988892 781.059082 0.519311 00:24
76 1598.570190 1943.324585 1936.156006 6.982880 0.001308 0.717146 -0.740403 5724.291016 0.948234 881.817749 770.956482 0.515682 00:23
77 1596.463501 1858.350708 1851.585205 6.690332 0.002513 0.700747 -0.784782 5736.254395 0.925605 861.278259 755.052063 0.510597 00:24
78 1591.501099 2161.671875 2154.538330 6.567418 -0.001495 0.686647 -0.821709 7697.672363 0.912659 860.494751 748.370300 0.496979 00:24
79 1596.576050 1848.325806 1841.201050 6.923673 -0.007202 0.719312 -0.734420 5649.215820 0.949537 881.026062 771.243774 0.515774 00:23
80 1578.273560 1850.652710 1843.813721 6.662688 -0.002183 0.708437 -0.763810 5656.493652 0.932729 867.856689 758.068176 0.512917 00:23
81 1558.910889 1857.016357 1850.471191 6.282830 0.003750 0.694354 -0.801634 6025.386719 0.911618 850.710693 742.491760 0.504756 00:23
82 1566.326172 1877.583496 1870.850586 6.587173 -0.000365 0.712432 -0.751816 5813.219238 0.933901 868.544617 755.775269 0.509591 00:24
83 1558.945923 1796.186035 1788.957031 7.010788 0.008110 0.711451 -0.757494 5445.705566 0.945560 876.531738 773.538818 0.519563 00:24
84 1563.117065 2025.193726 2015.877319 8.986026 0.001242 0.749838 -0.658024 5764.464355 1.027073 952.846375 855.554688 0.537086 00:24
85 1562.327148 1820.796997 1813.988525 6.611163 0.000196 0.713180 -0.751190 5506.539551 0.936482 867.845459 759.705750 0.513956 00:23
86 1544.917236 1775.630371 1768.784912 6.705034 0.005688 0.707357 -0.769205 5401.282227 0.933454 863.741150 762.567932 0.519356 00:24
87 1546.165283 1824.941895 1817.632690 7.065378 0.001202 0.721854 -0.730349 5397.050293 0.955980 882.884644 780.678223 0.524164 00:23
88 1546.792969 1851.563965 1844.506348 6.900307 0.006732 0.710861 -0.760693 5496.184570 0.942076 870.337891 771.701538 0.523635 00:24
89 1533.520386 1848.635742 1840.505615 8.000067 0.002263 0.733689 -0.702156 5597.409668 0.991777 915.240845 821.559875 0.538800 00:24
90 1533.974854 2042.176758 2033.418335 8.438148 -0.000659 0.734213 -0.701345 5859.502441 1.004081 928.179810 837.471558 0.540214 00:24
91 1534.598755 1770.096680 1762.933228 6.965265 0.000781 0.721685 -0.730158 5302.961426 0.953490 880.158203 776.901062 0.521827 00:24
92 1529.138550 1769.280640 1762.805542 6.223247 0.000261 0.700388 -0.789683 5462.433594 0.917643 848.088928 750.385254 0.521262 00:23
93 1514.251465 1756.898438 1750.141479 6.658386 -0.001312 0.703789 -0.780196 5348.198242 0.930945 860.246765 763.566895 0.522191 00:24
94 1511.639648 1742.198242 1735.486206 6.530146 0.004555 0.707732 -0.769576 5264.849121 0.932178 860.377258 762.482971 0.523589 00:24
95 1518.488770 1809.066895 1802.725586 6.125498 0.004147 0.690996 -0.813873 5853.875488 0.904323 839.218445 739.848206 0.511833 00:24
96 1506.064941 1757.221313 1750.776855 6.236708 -0.000482 0.703213 -0.782113 5332.096680 0.919954 849.483337 751.440063 0.521793 00:23
97 1492.611450 1749.808716 1743.591553 6.031801 0.008199 0.693211 -0.808296 5335.791016 0.903719 836.115479 738.152649 0.514672 00:24
98 1495.907227 1780.331055 1774.109863 5.997238 0.006235 0.682826 -0.837528 5565.546875 0.893420 827.784546 734.499817 0.511417 00:23
99 1506.924927 1753.827026 1747.724487 5.929687 0.008335 0.687965 -0.823548 5548.875977 0.897357 830.657043 735.534119 0.514324 00:24
100 1490.491211 1772.699097 1765.414429 7.126364 0.005110 0.712210 -0.759742 5342.322754 0.951754 876.143677 785.646484 0.531999 00:24
101 1490.735229 2094.084229 2087.595459 6.007836 0.017847 0.672853 -0.859687 7370.563965 0.883705 832.604309 725.148865 0.486253 00:24
102 1474.431885 1886.980835 1880.927002 5.964793 0.002663 0.685117 -0.831594 5843.044922 0.893664 828.043518 733.632629 0.513065 00:23
103 1488.557861 1763.483521 1757.162354 6.224775 0.003430 0.689050 -0.823036 5342.456055 0.904692 833.983154 745.886536 0.522442 00:23
104 1478.421631 1721.488403 1714.755249 6.576198 0.002086 0.705786 -0.775474 5198.592285 0.929646 855.405090 761.574036 0.523895 00:24
105 1477.666626 1734.629395 1728.487549 5.992128 -0.000108 0.690119 -0.818457 5326.170410 0.900211 829.373230 738.171143 0.517729 00:24
106 1473.125854 1963.250122 1954.735840 8.192457 -0.009232 0.725935 -0.725695 5698.272461 0.992598 912.751648 830.933044 0.544388 00:23
107 1470.645142 1754.682129 1748.505615 6.028650 -0.003051 0.689379 -0.821751 5312.978516 0.901407 830.095337 741.401245 0.522096 00:23
108 1466.332764 1757.086548 1750.146240 6.846844 -0.000674 0.705047 -0.780684 5243.905273 0.935956 859.592285 773.074219 0.533158 00:23
109 1469.749634 1726.563477 1720.450806 5.911416 0.003956 0.687264 -0.826549 5249.537598 0.896469 825.995483 736.348145 0.517084 00:23
110 1464.484375 1736.870850 1730.732300 5.962198 0.004571 0.686533 -0.828575 5334.952637 0.897237 826.512878 737.748352 0.516293 00:24
111 1474.863647 1793.794189 1788.000122 5.671723 -0.001588 0.674181 -0.861681 5883.505371 0.876902 814.074036 722.494751 0.505507 00:24
112 1462.521729 1711.135254 1704.977295 6.024343 0.000763 0.687511 -0.827819 5293.791016 0.900351 829.086060 742.556763 0.523178 00:24
113 1449.825928 1701.789673 1695.721924 5.878519 0.001709 0.687487 -0.828042 5216.040527 0.896621 825.075745 738.509460 0.524055 00:24
114 1448.282471 1792.912231 1785.387207 7.269108 0.003752 0.716244 -0.750997 5364.237305 0.961296 881.468506 797.154053 0.539143 00:24
115 1445.574585 1739.160400 1733.344604 5.582691 0.000341 0.675874 -0.857014 5535.911621 0.877061 811.681335 721.563904 0.506585 00:24
116 1447.305664 1702.951294 1696.921875 5.905175 -0.001069 0.685150 -0.835695 5261.783203 0.895340 821.455139 740.116638 0.526518 00:24
117 1440.761230 1798.639282 1791.698853 6.732737 0.006422 0.698484 -0.798006 5357.394531 0.930286 853.307495 770.295471 0.528635 00:24
118 1436.066528 1683.260254 1676.651978 6.451030 0.004302 0.699094 -0.796131 5083.051270 0.921508 844.567322 759.091492 0.528947 00:24
119 1435.049683 1834.709106 1827.218750 7.215334 0.002182 0.712799 -0.759363 5444.700195 0.956702 878.528870 792.816406 0.534509 00:23
120 1432.722534 1712.610962 1706.566895 5.798853 -0.005328 0.687854 -0.825481 5167.812500 0.894212 821.314392 734.109131 0.517869 00:23
121 1434.100586 1951.651611 1945.811279 5.471668 0.007534 0.673743 -0.859499 7142.018066 0.870716 811.510376 712.458984 0.492756 00:23
122 1437.508789 1674.697021 1668.088135 6.481055 -0.004835 0.701548 -0.789016 5056.125000 0.924078 847.228516 760.208435 0.528015 00:24
123 1434.666138 1735.760132 1730.124634 5.366279 0.007576 0.676784 -0.854084 5562.921875 0.872821 807.179810 715.587524 0.505585 00:24
124 1429.497192 1689.528076 1683.750244 5.527700 0.002926 0.677468 -0.853770 5230.196777 0.877476 808.014038 722.075500 0.511612 00:24
125 1418.321533 2098.251709 2087.811279 9.920640 0.010520 0.737638 -0.696218 5841.585449 1.045698 967.533997 895.270081 0.550583 00:23
126 1417.721924 1687.144531 1680.525635 6.484304 0.001014 0.700080 -0.794815 5142.148926 0.925255 845.612244 764.709595 0.532689 00:24
127 1415.999878 1661.122559 1654.787842 6.252984 -0.001776 0.690348 -0.821688 5054.491211 0.908621 830.860168 752.067505 0.529559 00:24
128 1419.004517 1668.103516 1661.842285 6.131657 0.004399 0.683106 -0.842830 5066.711914 0.900992 823.832336 749.567566 0.530140 00:24
129 1428.106201 1646.717651 1640.340942 6.264211 0.003743 0.692410 -0.817175 5054.361328 0.912708 833.564209 756.597290 0.533775 00:24
130 1412.550659 1658.395264 1652.331421 5.964541 0.005635 0.685173 -0.836093 5059.983887 0.897434 821.432312 743.021545 0.527028 00:24
131 1400.915649 1694.892700 1689.228394 5.458885 0.000171 0.670887 -0.872974 5170.104980 0.869605 799.230408 719.464050 0.509511 00:23
132 1415.313843 1657.355225 1650.954102 6.333950 -0.001637 0.689140 -0.826907 5043.656738 0.910786 832.046265 757.605103 0.534587 00:23
133 1410.529419 1861.309570 1853.625488 7.459136 0.000599 0.704333 -0.785786 5443.997070 0.955456 874.413452 800.895813 0.541768 00:24
134 1403.009766 1903.692871 1898.027344 5.320584 0.005749 0.659000 -0.902107 6834.922852 0.851381 795.139038 703.956909 0.486862 00:23
135 1399.830078 1701.127319 1695.625732 5.258708 0.001896 0.668509 -0.880920 5312.633301 0.862366 793.509644 714.375000 0.512364 00:24
136 1398.415649 1738.159668 1732.829834 5.186443 0.006218 0.673360 -0.863564 5501.898926 0.863448 798.265930 707.888245 0.502401 00:24
137 1399.475586 1891.271118 1885.845581 5.066854 0.005012 0.655829 -0.913119 6470.983398 0.843407 784.407654 700.131470 0.491701 00:24
138 1406.035889 1675.015747 1669.492188 5.342115 0.003427 0.678883 -0.849478 5177.966797 0.872906 802.791809 715.483704 0.510385 00:23
139 1396.128906 1638.963135 1633.346924 5.520921 0.002290 0.671443 -0.874290 5054.814453 0.871630 797.175049 724.165344 0.518922 00:24
140 1389.001709 1951.423584 1945.744263 5.251068 0.002636 0.661194 -0.898086 6788.769043 0.854937 794.803528 708.418152 0.495433 00:24
141 1388.087158 1688.355103 1682.019043 6.212550 -0.000564 0.681996 -0.846479 5130.877441 0.901388 823.665833 751.761047 0.530002 00:23
142 1385.485352 1639.908691 1633.910156 5.914208 0.000989 0.685338 -0.834779 5048.670410 0.895893 819.125366 740.246887 0.524021 00:24
143 1389.624268 1728.018799 1722.430908 5.454906 -0.002389 0.675913 -0.859976 5214.032715 0.873538 800.997437 720.956177 0.515619 00:23
144 1387.876587 1627.204834 1621.583496 5.461523 0.002939 0.666704 -0.888658 5018.274414 0.867548 794.538818 724.534485 0.519912 00:24
145 1395.786377 1799.681152 1794.277466 5.198103 0.004318 0.672103 -0.868977 5674.524414 0.863514 795.230957 710.991943 0.507598 00:23
146 1389.711426 2341.362305 2330.410645 10.206139 0.009791 0.737460 -0.699181 6225.690918 1.055980 972.888184 912.047424 0.557112 00:24
147 1382.315186 1889.569336 1882.606689 6.672486 0.002923 0.696264 -0.806550 5563.912109 0.928896 847.870911 773.047607 0.534387 00:23
148 1381.061890 1683.992554 1678.705200 5.088257 -0.000407 0.666899 -0.882866 5115.092285 0.854922 787.304565 704.907288 0.501818 00:24
149 1381.243164 1756.639404 1751.480225 5.025988 -0.004411 0.664984 -0.886566 5821.260254 0.849539 785.729736 698.703003 0.494932 00:24
150 1391.396118 1629.302002 1623.264282 5.902591 0.003420 0.686533 -0.832479 4975.734863 0.897541 818.911499 742.386169 0.527404 00:23
151 1380.467407 1841.547485 1836.219360 4.929632 0.006568 0.659674 -0.899792 6095.972168 0.843260 786.319092 694.334778 0.485277 00:23
152 1376.042358 1656.519165 1650.619995 5.746026 -0.001170 0.687702 -0.827185 5076.075684 0.892985 817.150696 734.218079 0.520622 00:24
153 1368.910645 1604.249878 1598.579346 5.557160 0.005134 0.675994 -0.862590 4929.545898 0.878161 801.331299 729.264526 0.524311 00:24
154 1364.348389 1711.368652 1705.100952 6.029387 0.007552 0.679768 -0.852199 5120.926270 0.894755 816.590027 745.362122 0.526926 00:24
155 1364.940796 1623.401489 1618.064819 5.157538 0.002106 0.669850 -0.877013 5024.353027 0.860458 788.950806 711.364502 0.511635 00:23
156 1369.497925 1737.252075 1732.073120 5.001041 -0.000255 0.660906 -0.899281 5684.628906 0.846304 782.234314 699.924072 0.495823 00:23
157 1375.073242 1670.945557 1665.610962 5.146222 0.000239 0.675878 -0.859120 5231.878418 0.865907 794.046570 711.487976 0.511762 00:23
158 1361.945435 1659.720459 1653.842285 5.687163 0.004606 0.677694 -0.857958 5073.202637 0.884437 807.569641 735.100952 0.525402 00:24
159 1359.260254 1672.385132 1667.300415 4.952787 0.002986 0.662565 -0.895641 5178.009766 0.846214 779.041077 699.466980 0.500852 00:24
160 1349.149414 1640.890137 1634.643066 6.153680 0.003964 0.684842 -0.839263 5043.590820 0.904352 822.992798 753.826355 0.532841 00:23
161 1353.739624 1597.526001 1592.349854 5.021693 0.006057 0.660444 -0.905429 5003.083008 0.848760 777.452271 707.987366 0.511559 00:23
162 1347.540649 1638.532959 1633.320190 5.058012 0.000977 0.651229 -0.934468 5096.089844 0.841315 770.387146 710.256409 0.515415 00:24
163 1335.703491 1695.615967 1689.228760 6.245860 0.005231 0.682442 -0.846417 5214.112793 0.904372 825.082397 756.296143 0.532854 00:24
164 1346.623779 1590.747314 1585.051636 5.653391 0.000028 0.677425 -0.858526 4957.563965 0.882757 804.979065 733.336365 0.524897 00:24
165 1333.729492 1623.744019 1618.688232 4.874943 0.000402 0.646668 -0.947792 5077.589355 0.832516 762.127869 704.936768 0.512799 00:23
166 1333.856934 1700.821411 1695.920654 4.680223 0.000258 0.645109 -0.949244 5450.968750 0.823805 758.155823 694.016968 0.501195 00:24
167 1327.820557 1576.457520 1571.300537 5.078076 0.001820 0.654918 -0.923109 4919.488770 0.844439 772.418518 709.839233 0.514651 00:24
168 1328.926758 1573.758301 1568.447388 5.132746 0.002853 0.656624 -0.919550 4844.415527 0.850984 776.917114 716.910889 0.520079 00:24
169 1319.658813 1629.252441 1623.602295 5.531579 0.004089 0.663933 -0.900344 4970.062988 0.868671 791.313416 732.179138 0.529097 00:24
170 1311.670044 1567.386597 1562.363037 5.011216 -0.002171 0.646587 -0.949217 4992.060547 0.834543 762.800232 708.426880 0.516315 00:24
171 1311.979126 1565.485718 1560.121094 5.253644 0.003709 0.652157 -0.933970 4863.855469 0.849615 774.268250 720.804810 0.522821 00:24
172 1309.738159 1569.579224 1564.234375 5.176810 -0.001438 0.645019 -0.957222 4800.472168 0.841719 767.224060 721.150696 0.526419 00:24
173 1300.042969 1576.110474 1571.340088 4.693328 0.002788 0.635771 -0.981228 4908.370117 0.815928 747.155457 698.806824 0.510915 00:24
174 1300.506714 1596.154785 1591.422485 4.600417 0.000205 0.631884 -0.993400 5049.825195 0.811167 743.685730 698.102051 0.510750 00:24
175 1302.210083 1565.038940 1560.224731 4.715932 -0.002279 0.632513 -0.993281 4823.053223 0.815410 744.632507 703.829956 0.516547 00:24
176 1295.195312 1634.335571 1628.611816 5.639289 0.004621 0.643383 -0.965427 5045.840820 0.854679 777.336243 740.196472 0.536837 00:24
177 1296.105835 1552.511475 1547.794678 4.642930 -0.001359 0.625849 -1.015444 4811.358887 0.807235 736.801025 703.913940 0.519581 00:23
178 1289.839111 1555.052490 1550.240967 4.691702 0.002196 0.623276 -1.024005 4831.184082 0.807397 736.174866 707.429565 0.520973 00:23
179 1288.338867 1675.798340 1669.824585 5.750722 0.008604 0.646306 -0.958059 5084.871582 0.863302 785.137085 748.276550 0.542142 00:23
180 1283.673828 1516.768433 1511.867798 4.808212 0.000827 0.625714 -1.017968 4717.286621 0.813187 739.968689 712.312256 0.526445 00:24
181 1276.340210 1549.146240 1544.537231 4.559086 -0.000139 0.611956 -1.059541 4878.202148 0.791074 722.302429 703.156128 0.517453 00:24
182 1275.160889 1523.992432 1519.262573 4.653107 0.003621 0.616803 -1.046131 4775.168457 0.800713 729.311401 709.372437 0.525189 00:24
183 1274.201660 1535.204346 1530.678711 4.456763 0.004054 0.606965 -1.076548 4829.563477 0.784311 716.280945 703.056152 0.519575 00:23
184 1268.209961 1511.440186 1506.698242 4.603409 0.004325 0.610509 -1.067291 4754.063965 0.794557 723.420471 711.402893 0.527470 00:24
185 1266.569946 1543.884888 1539.392822 4.316476 0.002206 0.603420 -1.087745 4918.042969 0.778405 711.360107 701.144226 0.517730 00:24
186 1263.202515 1528.685181 1523.815796 4.723829 0.001786 0.614226 -1.057132 4734.013184 0.803246 729.756042 717.827942 0.533173 00:23
187 1260.967529 1508.121826 1503.532715 4.507143 0.000652 0.605073 -1.085063 4768.585449 0.786481 716.546814 709.896118 0.526683 00:24
188 1254.141235 1521.172729 1516.287964 4.772686 0.003552 0.610881 -1.069029 4713.490723 0.802140 728.285034 721.653503 0.536562 00:24
189 1258.827637 1500.663452 1496.237183 4.378232 0.001310 0.597949 -1.108888 4722.563965 0.776632 706.947815 709.265503 0.527378 00:24
190 1254.742188 1494.618530 1490.014893 4.517956 0.002078 0.599058 -1.107524 4685.872070 0.783946 711.739197 717.455383 0.534846 00:23
191 1253.978394 1573.777710 1569.478027 4.117863 0.000867 0.586330 -1.145558 5067.717773 0.757283 692.494629 703.100769 0.519190 00:23
192 1248.770020 1492.288940 1487.655640 4.550579 0.001854 0.598468 -1.109971 4670.904785 0.784323 711.917358 719.044434 0.536398 00:24
193 1247.504150 1494.558472 1489.838989 4.692032 0.002832 0.601183 -1.101408 4682.655273 0.789145 715.769653 720.914001 0.537973 00:24
194 1246.692139 1497.859863 1493.126465 4.654514 0.002181 0.602137 -1.098455 4664.011719 0.791181 717.602112 721.982056 0.538484 00:24
195 1248.151001 1487.668579 1483.009277 4.578956 0.002568 0.598033 -1.111644 4664.344238 0.783169 710.659485 718.671692 0.536958 00:24
196 1243.887207 1486.362183 1481.893433 4.425810 0.001585 0.593358 -1.126205 4675.741699 0.774591 703.536987 715.078613 0.533562 00:24
197 1238.384277 1485.278931 1480.723999 4.445567 0.001731 0.592773 -1.128412 4670.096680 0.774399 703.236633 715.891174 0.534319 00:24
198 1243.141846 1485.222656 1480.728638 4.453456 0.002115 0.593524 -1.125854 4668.957031 0.775106 703.962830 715.587891 0.534109 00:24
199 1240.411377 1485.554810 1481.032349 4.460090 0.001861 0.592433 -1.129834 4674.369629 0.774331 703.022034 716.537415 0.535203 00:24
{% endraw %} {% raw %}
prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')
Path('models/frozen-MMDVae-TMP-latent128-resblock-alpha10_2021-03-24_10.15.31.pth')
{% endraw %}

128 latents, alpha=20

{% raw %}
latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 20
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
     
    
    
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.33113112449646,
 0.04786301031708717,
 0.18949706740677358,
 0.1258925348520279)
{% endraw %} {% raw %}
#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mmd mu std logvar l1_b_mean mu_sd l1_latent_reg weighted_kld logvar_sd time
0 44869.117188 38630.105469 38172.390625 452.347382 0.363450 2.003438 1.154238 49319.636719 4.148453 8080.807617 26520.181641 0.966020 00:23
1 28217.714844 19543.214844 19365.693359 173.174393 0.211772 1.590892 0.723334 34397.785156 2.722190 5338.421875 11395.580078 0.892636 00:23
2 18315.355469 13255.430664 13121.753906 130.631409 0.219888 1.532194 0.665691 27403.474609 2.465842 4917.343750 9348.974609 0.835872 00:23
3 12825.623047 9738.818359 9650.188477 86.012886 0.174977 1.367238 0.465474 22967.119141 2.081707 4158.813965 6535.408691 0.760577 00:24
4 9639.348633 7791.971191 7737.542480 52.770176 0.111142 1.187918 0.203920 20011.353516 1.711145 3402.236816 4313.964844 0.716786 00:24
5 7687.958496 6969.908203 6936.609375 31.604336 0.071600 1.017000 -0.096666 18651.474609 1.397718 2771.013672 2869.612793 0.689790 00:24
6 6420.611328 5956.696289 5929.208008 25.928036 0.055198 0.933785 -0.259582 16708.939453 1.273471 2517.689209 2402.669678 0.671029 00:24
7 5523.093750 5197.247559 5173.945801 21.902893 0.056021 0.890996 -0.346857 14908.278320 1.195464 2358.052979 2146.462646 0.652390 00:23
8 4918.751465 4908.517578 4887.394043 19.962851 0.048813 0.845242 -0.446389 13966.420898 1.135524 2239.118164 1974.152588 0.633212 00:24
9 4473.037598 4544.405762 4524.976074 18.338888 0.045379 0.825765 -0.490280 13264.149414 1.100679 2152.622803 1881.080322 0.627626 00:23
10 4142.837402 4177.356934 4157.032715 19.363203 0.038059 0.848681 -0.432753 12074.405273 1.130283 2209.923340 1944.031250 0.618873 00:24
11 3917.388916 3997.903320 3977.184814 19.701015 0.057482 0.848224 -0.430466 11531.174805 1.133727 2215.214111 1949.137207 0.608181 00:24
12 3699.811768 3793.642334 3773.311523 19.624884 0.058232 0.857292 -0.405696 10980.250000 1.137904 2227.622314 1948.683960 0.595239 00:24
13 3548.590576 3717.894043 3695.793945 21.325510 0.051739 0.856475 -0.406966 10586.673828 1.154981 2252.016113 1997.359985 0.593199 00:24
14 3392.223633 3564.068848 3543.844238 19.469976 0.048081 0.859627 -0.396350 10151.842773 1.136936 2212.456299 1934.764526 0.582036 00:24
15 3294.741943 3458.957520 3437.580811 20.652731 0.035036 0.864861 -0.385641 9724.040039 1.155857 2242.773926 1989.750732 0.587111 00:24
16 3188.714355 3471.833740 3450.045410 21.072681 0.061831 0.877749 -0.353019 9386.150391 1.169560 2276.136963 2020.442383 0.577378 00:24
17 3111.545898 3386.387207 3364.288574 21.481207 0.057307 0.876575 -0.355280 9266.531250 1.172527 2275.385742 2028.340576 0.576368 00:24
18 3067.808838 3286.744629 3263.543457 22.496641 0.058035 0.901485 -0.299061 8925.688477 1.206159 2334.083984 2121.989258 0.577001 00:23
19 3007.293945 3189.976807 3167.430908 21.941383 0.049392 0.884795 -0.334096 8704.542969 1.184379 2293.989990 2053.986328 0.570027 00:23
20 2945.777100 3182.058105 3159.409668 22.111326 0.049436 0.907009 -0.285508 8585.974609 1.205574 2330.225830 2113.958740 0.571989 00:23
21 2895.951416 3211.220459 3188.865967 21.689819 0.046961 0.886839 -0.327073 8588.288086 1.182075 2284.830322 2039.987183 0.562882 00:24
22 2849.264160 3051.117188 3028.877930 21.914141 0.062428 0.886953 -0.327097 8263.973633 1.183866 2291.922607 2047.876343 0.563708 00:24
23 2808.139648 3054.230713 3032.758301 20.943661 0.052907 0.880357 -0.340083 8256.331055 1.169980 2262.256104 2002.924072 0.556725 00:24
24 2773.731201 3043.178711 3019.842285 22.779768 0.056994 0.902545 -0.291591 8085.221680 1.206399 2338.058105 2110.115967 0.558648 00:23
25 2742.290527 2963.630859 2941.413818 21.570345 0.058292 0.876817 -0.347950 7992.745117 1.172185 2268.051758 2011.426392 0.555742 00:23
26 2719.287354 2948.922852 2925.617676 22.668314 0.061004 0.901012 -0.293574 7791.686523 1.204079 2328.265869 2100.627441 0.555158 00:23
27 2679.970215 2916.285645 2894.107422 21.886332 0.058098 0.897683 -0.300988 7793.752441 1.191654 2299.196289 2063.407471 0.554300 00:23
28 2657.288818 2888.732910 2867.198486 21.124582 0.054443 0.885014 -0.327630 7721.035645 1.174288 2261.996826 2010.303223 0.548887 00:23
29 2637.341553 2875.486816 2852.899658 22.185432 0.052402 0.899004 -0.297901 7635.975098 1.196017 2309.848877 2075.057373 0.552954 00:24
30 2611.630371 2916.487549 2893.916016 22.165520 0.060199 0.892029 -0.311832 7624.857910 1.190608 2294.882568 2058.446533 0.548067 00:23
31 2601.701660 2958.729736 2937.280029 21.088247 0.052859 0.875972 -0.348007 7780.184570 1.164899 2237.513916 1986.447510 0.547276 00:24
32 2560.690186 2890.773193 2868.005615 22.291035 0.053579 0.893000 -0.311326 7511.469238 1.193920 2301.701660 2071.535645 0.552681 00:24
33 2536.779785 2881.475586 2859.619873 21.348116 0.060162 0.896214 -0.303677 7471.465332 1.185748 2284.387695 2045.750488 0.550699 00:24
34 2520.879150 2919.730957 2896.383057 22.488171 0.056736 0.900711 -0.294199 7453.818848 1.202283 2313.757324 2095.693848 0.553407 00:23
35 2495.342773 2774.852539 2752.407227 21.960470 0.051357 0.895464 -0.304288 7264.982422 1.192231 2286.324463 2062.578125 0.546508 00:23
36 2484.418701 2731.136230 2710.085938 20.814085 0.050483 0.880643 -0.337369 7257.747559 1.166030 2236.843018 1987.916504 0.546926 00:23
37 2467.477051 2689.622559 2667.826416 21.342794 0.057822 0.896463 -0.301654 7114.496094 1.185970 2275.238037 2043.145142 0.544947 00:23
38 2457.845703 2991.763428 2970.408203 21.072306 0.052153 0.889041 -0.318461 7656.958008 1.176745 2257.115234 2017.733765 0.545500 00:24
39 2322.232178 2856.982422 2834.868896 21.692192 0.040446 0.881323 -0.335423 7360.048828 1.178361 2268.504395 2023.965210 0.540828 00:24
40 2227.645508 2579.064941 2557.928467 20.654335 0.044434 0.849737 -0.406745 6973.835449 1.140613 2198.822021 1924.729004 0.536686 00:24
41 2180.507080 2692.549805 2674.151611 18.129353 0.046520 0.833835 -0.442311 7177.255859 1.097045 2103.624512 1805.527344 0.531004 00:24
42 2126.195068 2674.731445 2655.945312 18.241838 0.050435 0.825340 -0.461466 7156.921875 1.091575 2089.953613 1794.163574 0.526470 00:23
43 2091.339111 2310.899902 2293.354248 17.364128 0.047284 0.810492 -0.496608 6644.475098 1.069493 2046.097046 1742.156250 0.522600 00:24
44 2053.124756 2347.489014 2329.062744 18.080427 0.043661 0.815187 -0.483483 6631.128418 1.079206 2075.353027 1761.062134 0.515697 00:24
45 2039.703735 2485.505127 2466.542236 18.650936 0.058622 0.830873 -0.447020 6744.937988 1.100019 2106.931396 1812.271118 0.521363 00:24
46 2014.683228 2239.049316 2220.724609 18.121702 0.051046 0.826565 -0.456548 6346.936035 1.089885 2093.416748 1784.469604 0.516715 00:24
47 1984.203735 2245.293701 2228.103271 16.879969 0.047386 0.808704 -0.498042 6430.269531 1.061489 2030.824341 1715.867310 0.510975 00:24
48 1979.119629 2352.370117 2333.208984 18.823326 0.047247 0.831529 -0.445145 6513.161133 1.102933 2110.088379 1818.141846 0.519061 00:23
49 1951.241577 2186.123779 2168.409912 17.496885 0.047719 0.818824 -0.473539 6246.245605 1.074579 2055.149414 1743.373291 0.510603 00:24
50 1928.953369 2226.338623 2209.330811 16.821451 0.041958 0.804572 -0.508660 6426.308594 1.056968 2022.071777 1707.827515 0.509897 00:23
51 1918.752075 2166.478516 2149.046387 17.217678 0.049111 0.804747 -0.509535 6236.624023 1.061681 2028.734375 1724.478760 0.514899 00:23
52 1903.347168 2149.011719 2130.229492 18.649336 0.054450 0.823058 -0.465431 6080.879883 1.092651 2093.164307 1797.216919 0.515208 00:24
53 1888.991211 2195.260742 2177.610352 17.639759 0.044792 0.817150 -0.477495 6222.992676 1.076117 2056.481201 1749.180542 0.507933 00:24
54 1881.612183 2141.744629 2124.793945 16.878689 0.047793 0.809202 -0.495584 6120.611816 1.057818 2021.609009 1703.235962 0.503281 00:23
55 1854.816528 2118.498291 2101.327881 17.104313 0.051836 0.810419 -0.493421 6000.839355 1.062874 2028.957764 1718.250610 0.505966 00:24
56 1839.432861 2258.764648 2241.761963 16.793386 0.044510 0.800588 -0.517331 6347.492676 1.051791 2010.334839 1695.131836 0.504905 00:24
57 1853.467041 2232.258545 2212.551025 19.500610 0.045772 0.834733 -0.436135 6296.918945 1.113015 2137.290771 1842.341064 0.510965 00:24
58 1837.273560 2078.458740 2061.684082 16.776508 0.045135 0.804238 -0.508361 6023.635742 1.054307 2008.124268 1699.466187 0.504561 00:24
59 1816.862427 2040.441772 2024.711670 15.597315 0.045059 0.785856 -0.552272 6029.678223 1.024677 1958.124878 1633.727783 0.496716 00:24
60 1819.208252 2139.017090 2121.553223 17.336550 0.055248 0.818801 -0.471689 6070.657227 1.071967 2050.424561 1734.173218 0.500660 00:23
61 1799.726440 2094.442139 2078.879150 15.395780 0.047046 0.782574 -0.561578 6207.199219 1.019686 1940.321533 1627.035156 0.499807 00:24
62 1783.477295 2189.658203 2173.854248 15.552533 0.050241 0.781507 -0.563103 6475.451172 1.019956 1941.290161 1626.447998 0.495016 00:24
63 1768.642334 2078.535889 2062.391846 15.885236 0.046041 0.790759 -0.541188 5978.875977 1.034647 1967.154175 1658.364868 0.500458 00:23
64 1764.174438 2000.245361 1984.001587 16.078659 0.050129 0.793963 -0.532840 5841.421875 1.038195 1977.391968 1664.403687 0.498570 00:24
65 1755.285645 2074.506592 2057.000977 17.236784 0.045309 0.807654 -0.500120 5898.439453 1.064845 2027.080688 1725.923340 0.503585 00:24
66 1751.253052 2192.845703 2176.969482 15.646471 0.047620 0.794049 -0.531214 6264.810547 1.032213 1964.389160 1644.656860 0.494554 00:24
67 1742.702148 2003.918945 1988.279419 15.603477 0.048556 0.792067 -0.536823 5857.966309 1.029384 1960.994385 1640.948608 0.496105 00:23
68 1734.003662 1960.304810 1944.389404 15.780478 0.042396 0.783507 -0.559525 5773.011719 1.025699 1953.704346 1641.869385 0.499045 00:23
69 1732.676270 2056.047363 2037.766235 17.803648 0.048374 0.805790 -0.505154 5835.386230 1.070438 2045.084351 1744.775269 0.504155 00:23
70 1719.732910 1968.918945 1953.201294 15.736311 0.049873 0.781208 -0.565390 5756.400391 1.023440 1944.613037 1639.710449 0.500073 00:24
71 1722.252686 2068.263672 2050.161621 17.727818 0.049945 0.812661 -0.488522 5823.361816 1.074614 2049.888184 1750.921021 0.505066 00:24
72 1715.597046 1943.568359 1928.785889 14.659951 0.044364 0.782311 -0.560319 5696.813965 1.011397 1924.081909 1601.036621 0.492042 00:23
73 1706.087524 2009.766113 1993.419189 16.229599 0.052192 0.795711 -0.528156 5771.199219 1.040488 1984.187622 1668.851074 0.497337 00:24
74 1690.334961 2119.673584 2104.496582 14.836791 0.047608 0.783031 -0.558614 6084.231445 1.013145 1926.117798 1605.451904 0.491907 00:24
75 1683.371094 1925.619385 1910.288330 15.424359 0.053724 0.778009 -0.573182 5706.276855 1.016423 1933.619019 1624.667114 0.498465 00:23
76 1671.691162 1938.888306 1923.586182 15.093276 0.047824 0.783145 -0.559653 5705.539551 1.016528 1929.590210 1617.487305 0.496029 00:23
77 1675.195557 1969.706665 1952.145996 17.220224 0.050449 0.801974 -0.512465 5722.925293 1.058287 2022.044189 1710.370239 0.497038 00:23
78 1667.553467 1951.343872 1935.015015 16.264315 0.045767 0.795914 -0.528531 5608.801270 1.041058 1982.651611 1671.711548 0.500002 00:23
79 1666.695801 1924.638428 1909.485229 14.979011 0.048244 0.778510 -0.570834 5615.051758 1.011054 1921.656250 1606.899292 0.493661 00:23
80 1667.953369 1927.399292 1912.782471 14.434852 0.048317 0.773981 -0.579921 5788.229492 1.000179 1900.219604 1578.418213 0.485158 00:23
81 1656.584839 1962.813721 1946.891479 15.511126 0.044743 0.791771 -0.538538 5702.780273 1.029768 1956.088745 1644.844849 0.498330 00:23
82 1637.365845 1897.853760 1882.795776 15.077114 0.045206 0.779908 -0.568294 5562.270996 1.011593 1918.007202 1609.076904 0.497358 00:23
83 1623.953735 1895.878540 1880.903564 14.739231 0.040865 0.766553 -0.603555 5562.432129 0.999141 1893.257202 1593.663086 0.499841 00:24
84 1631.195312 2004.035767 1989.149414 14.538733 0.047065 0.778626 -0.570778 5742.729004 1.006706 1907.639526 1596.172241 0.495295 00:23
85 1628.239136 1901.397583 1887.269165 13.971234 0.036782 0.762744 -0.610812 5755.364746 0.984519 1865.953613 1554.630371 0.490754 00:24
86 1635.655884 1890.822021 1876.614990 13.997304 0.047380 0.768836 -0.594105 5630.626465 0.990158 1881.327148 1560.892212 0.487769 00:23
87 1619.503906 1882.146240 1867.116211 14.936031 0.045019 0.769721 -0.595086 5506.448242 1.005221 1908.795898 1605.485474 0.498761 00:23
88 1619.595337 1880.541260 1865.143188 15.202730 0.049003 0.787871 -0.548113 5462.920898 1.022931 1941.532959 1630.943115 0.496960 00:23
89 1613.255249 1984.275024 1966.732910 17.282715 0.042405 0.804184 -0.508741 5638.116211 1.062198 2025.288452 1723.226807 0.501592 00:24
90 1606.169189 1862.778687 1847.071533 15.529962 0.044467 0.783225 -0.560873 5462.847656 1.022386 1939.350952 1636.303101 0.500584 00:23
91 1609.607544 1910.367065 1894.708618 15.496771 0.044426 0.790351 -0.541698 5494.456543 1.027497 1953.776001 1639.620239 0.496587 00:24
92 1603.590942 1960.099365 1946.069580 13.971668 0.046126 0.754995 -0.630675 5919.393555 0.977296 1855.564209 1545.912354 0.489731 00:24
93 1605.382080 1876.894043 1862.129883 14.676646 0.040245 0.775969 -0.576931 5583.471680 1.006059 1906.842041 1595.229858 0.492825 00:24
94 1583.398438 1937.457764 1921.093994 16.335154 0.041187 0.791778 -0.540101 5531.172852 1.039418 1977.000122 1674.374512 0.503145 00:23
95 1581.612549 1835.172241 1821.100098 13.890847 0.050193 0.765758 -0.603429 5450.705078 0.986337 1868.951904 1558.364624 0.491370 00:24
96 1579.690430 1837.527100 1822.542603 14.684216 0.048580 0.774344 -0.583328 5418.464355 1.004971 1905.832397 1600.656006 0.499638 00:24
97 1583.773926 1821.455322 1807.557617 13.652730 0.046067 0.761249 -0.615050 5497.828125 0.981084 1855.827515 1549.596802 0.491711 00:24
98 1583.227417 1819.533691 1804.496216 14.928608 0.041859 0.781699 -0.563601 5330.727051 1.012702 1914.940430 1609.623901 0.496521 00:23
99 1567.252808 1902.686035 1886.798462 15.758029 0.050320 0.780095 -0.570298 5583.282227 1.022527 1943.718628 1644.271240 0.505894 00:24
100 1572.749634 1863.321655 1849.039185 13.978075 0.041848 0.776276 -0.576076 5542.300293 0.998581 1885.867798 1576.105713 0.490765 00:24
101 1561.390869 1778.629150 1765.033936 13.473954 0.046851 0.760431 -0.617726 5365.498535 0.977076 1847.461060 1542.018799 0.493202 00:24
102 1554.405762 1793.989136 1780.624512 13.337927 0.039644 0.762962 -0.609675 5439.504883 0.976549 1845.786499 1533.581421 0.489440 00:24
103 1550.139282 1803.763306 1789.418335 14.204505 0.047718 0.777707 -0.573318 5340.282715 1.001259 1893.551270 1584.347534 0.495005 00:24
104 1545.177002 1808.277588 1794.598145 13.563501 0.045840 0.774189 -0.581362 5352.294434 0.989471 1872.578247 1555.489868 0.491507 00:23
105 1546.557739 1778.235474 1764.073730 14.111894 0.041205 0.778304 -0.571823 5251.641113 0.999497 1890.440552 1578.523682 0.494077 00:23
106 1537.222168 1781.041260 1767.304443 13.775725 0.041837 0.771026 -0.589410 5304.643066 0.988058 1865.651245 1554.898193 0.490515 00:24
107 1538.707764 1785.395874 1771.354004 13.960267 0.039857 0.780740 -0.565743 5323.761230 1.000003 1889.690552 1577.392700 0.494897 00:24
108 1533.052368 1833.327026 1819.516113 13.616963 0.046818 0.768385 -0.596994 5411.942383 0.986244 1861.312378 1555.800903 0.493678 00:23
109 1528.282227 1798.111816 1784.515137 13.528557 0.041809 0.762646 -0.611968 5357.743652 0.978249 1844.739990 1541.980103 0.493448 00:24
110 1526.862427 1799.446045 1786.277954 12.966899 0.043644 0.754512 -0.632367 5376.239258 0.964648 1818.849487 1516.334839 0.490012 00:24
111 1523.077637 1756.224487 1742.447998 13.529755 0.046487 0.771139 -0.589131 5274.266602 0.986443 1864.797241 1551.135742 0.490616 00:23
112 1529.325684 1797.935425 1784.176392 13.653956 0.040448 0.760077 -0.619595 5376.958984 0.978359 1844.459961 1547.438232 0.496279 00:24
113 1518.242676 1783.705078 1768.756226 14.822066 0.041936 0.775163 -0.582138 5282.527344 1.008120 1905.776001 1609.571045 0.501678 00:24
114 1520.407837 1912.749268 1897.477661 14.960381 0.051009 0.773271 -0.587717 5464.833496 1.008230 1902.993286 1614.846191 0.504838 00:24
115 1505.013062 1741.917847 1728.625122 13.149019 0.045469 0.759637 -0.619913 5271.270508 0.972376 1835.787842 1531.567139 0.493255 00:24
116 1509.686890 1916.331909 1900.899414 15.149902 0.051836 0.780137 -0.569676 5540.610352 1.016474 1923.016479 1627.815796 0.503041 00:24
117 1508.735962 1787.357178 1773.621582 13.524192 0.053513 0.768796 -0.595543 5362.060059 0.984265 1858.056152 1550.191162 0.492759 00:24
118 1501.052124 1791.610962 1778.476440 13.032609 0.045079 0.754121 -0.633713 5358.076660 0.965411 1822.041870 1519.513306 0.490795 00:24
119 1497.884155 1820.942505 1807.536865 13.198060 0.047576 0.752508 -0.639384 5370.798340 0.967550 1824.296631 1530.348755 0.496000 00:23
120 1499.869019 1731.496460 1718.068115 13.332547 0.045602 0.762194 -0.613499 5209.441895 0.977164 1840.784058 1541.098877 0.494337 00:24
121 1494.199707 1785.917114 1771.454224 14.315552 0.044095 0.782811 -0.561474 5258.367676 1.006539 1896.658691 1594.894897 0.498379 00:24
122 1484.143921 1752.403809 1739.713379 12.572518 0.042145 0.759958 -0.616125 5328.588867 0.963721 1820.042847 1502.684448 0.483027 00:24
123 1486.989014 1785.623657 1771.019897 14.418050 0.043136 0.774449 -0.583683 5241.328613 1.002226 1890.029541 1594.741089 0.501373 00:24
124 1491.107422 1840.187866 1827.505127 12.358881 0.037512 0.742803 -0.660973 5813.546387 0.946614 1792.139893 1481.054810 0.480716 00:24
125 1483.239990 1771.854858 1757.864868 13.799751 0.050438 0.777081 -0.574626 5345.870605 0.995722 1878.988159 1570.665894 0.493920 00:23
126 1480.210815 1763.597046 1750.544800 12.836942 0.038926 0.755063 -0.630958 5339.038574 0.963782 1815.727051 1513.101196 0.490371 00:24
127 1486.744751 1712.286255 1699.191284 13.088781 0.039298 0.765617 -0.604729 5259.208984 0.975931 1836.793945 1533.706909 0.493931 00:24
128 1471.546997 1744.941772 1731.547485 13.081504 0.039608 0.770892 -0.588619 5255.071289 0.980480 1847.018799 1533.461182 0.486240 00:24
129 1477.258301 1885.250488 1872.204834 12.834613 0.038478 0.752977 -0.632669 6258.566895 0.959195 1819.870117 1495.808105 0.476098 00:24
130 1476.063721 1722.854736 1709.984497 12.728988 0.046360 0.758009 -0.624227 5213.971680 0.966029 1821.071289 1518.164551 0.493540 00:23
131 1468.545532 1766.099487 1753.359375 12.661175 0.043363 0.762251 -0.613217 5275.878418 0.968005 1820.734619 1517.834229 0.493148 00:24
132 1460.417725 1691.528564 1678.043335 13.388208 0.044237 0.767331 -0.600559 5067.453613 0.981512 1850.208130 1547.028442 0.495345 00:24
133 1460.132812 1764.835205 1750.694824 13.836353 0.044856 0.778354 -0.572423 5155.833984 0.998214 1878.788574 1577.688232 0.496308 00:24
134 1449.852295 1717.159668 1703.481445 13.584293 0.045624 0.771395 -0.591638 5100.827148 0.988719 1863.547607 1564.609619 0.500363 00:24
135 1456.299805 1796.339844 1783.667358 12.612735 0.046080 0.750525 -0.642769 5302.807129 0.956472 1799.921875 1501.478394 0.488982 00:24
136 1452.066650 1712.855835 1700.749756 12.010898 0.035147 0.742716 -0.663375 5219.075195 0.942903 1773.968506 1477.056396 0.487644 00:24
137 1442.379883 1861.538940 1848.774536 12.546704 0.044301 0.757024 -0.625706 5465.744629 0.960394 1808.829956 1502.808716 0.489563 00:23
138 1451.181396 1740.815430 1727.772705 12.861029 0.046706 0.763278 -0.609869 5193.144531 0.970980 1828.928711 1522.987793 0.491339 00:24
139 1454.517456 1739.416748 1726.141968 13.127309 0.044694 0.763143 -0.611723 5189.341797 0.975750 1836.774536 1538.444946 0.495323 00:23
140 1453.098999 1809.774414 1796.948120 12.717151 0.037457 0.770682 -0.590425 5266.688965 0.975846 1836.441528 1524.875000 0.490269 00:24
141 1453.698975 1766.385376 1754.002197 12.168653 0.039576 0.760169 -0.614894 5354.198730 0.958532 1808.237793 1487.761963 0.480523 00:24
142 1436.513794 1740.355835 1727.632935 12.607018 0.044312 0.762749 -0.611572 5135.670410 0.967614 1823.881104 1515.742188 0.492147 00:24
143 1426.954102 1683.543213 1670.891602 12.386132 0.047308 0.758007 -0.623049 5045.751953 0.961070 1808.852173 1503.491089 0.489031 00:23
144 1435.731812 1737.902100 1725.594482 12.203060 0.038761 0.759287 -0.617784 5304.227539 0.958251 1806.407349 1489.541870 0.482690 00:23
145 1424.604614 1679.959961 1666.985229 13.009170 0.044784 0.765574 -0.604158 5029.961426 0.975828 1840.403076 1532.814331 0.491593 00:23
146 1430.846802 1745.726562 1733.709106 11.779364 0.041196 0.749969 -0.643417 5346.671875 0.946536 1781.426025 1475.560059 0.485592 00:24
147 1426.728638 1670.089111 1657.807495 12.069998 0.039616 0.753894 -0.633200 5110.452637 0.953304 1794.025757 1487.391113 0.486382 00:24
148 1421.082153 1770.327026 1758.205566 11.881845 0.031985 0.737488 -0.674429 5852.652344 0.935614 1771.917725 1459.603271 0.477787 00:24
149 1424.906250 1770.970581 1756.427368 14.251448 0.045018 0.778558 -0.571325 5154.751465 1.003317 1893.021362 1589.802979 0.494235 00:24
150 1428.417847 1722.283813 1709.760742 12.425946 0.044141 0.769665 -0.592743 5079.943359 0.971751 1825.433105 1515.940063 0.489371 00:24
151 1421.149902 1690.783936 1677.105225 13.628743 0.038097 0.767513 -0.600250 5135.013184 0.985277 1856.369751 1556.523438 0.495316 00:23
152 1433.208496 1708.456787 1696.220093 12.007226 0.042655 0.762878 -0.608496 5153.088867 0.959358 1806.983887 1488.737061 0.482379 00:24
153 1426.762085 1661.465698 1649.183472 12.254661 0.038874 0.767195 -0.597527 5165.588379 0.966164 1817.744263 1500.536743 0.483345 00:24
154 1413.873413 1764.028931 1751.354736 12.607476 0.041934 0.758214 -0.623549 5231.798828 0.962364 1810.543091 1508.350464 0.491851 00:24
155 1420.080811 1658.453003 1646.169556 12.203891 0.041853 0.761522 -0.614779 5024.637207 0.961713 1806.474243 1502.410034 0.492658 00:24
156 1413.686279 1679.697388 1667.682251 11.885459 0.037733 0.745325 -0.656726 5111.092773 0.943062 1773.028198 1475.005493 0.489388 00:24
157 1400.979492 1654.799316 1642.494995 12.223604 0.037701 0.755133 -0.631057 4976.752930 0.956361 1798.080322 1495.637329 0.489962 00:24
158 1401.699585 1685.698975 1673.149658 12.474750 0.043362 0.763767 -0.608483 5032.924805 0.966380 1817.423096 1510.497803 0.490456 00:24
159 1388.167847 1656.265991 1643.767822 12.446927 0.038312 0.761289 -0.615861 4995.916504 0.965072 1812.325317 1511.438354 0.493681 00:24
160 1384.202393 1649.645386 1637.485352 11.945498 0.044549 0.755672 -0.629659 4986.094238 0.953666 1792.926758 1489.143188 0.490317 00:23
161 1380.549316 1764.954346 1751.468750 13.289563 0.038319 0.755761 -0.630709 5191.483887 0.971387 1829.248657 1535.142456 0.494361 00:23
162 1375.546021 1743.503296 1732.010742 11.156135 0.041137 0.728400 -0.699738 5502.562500 0.918260 1733.994873 1433.800537 0.478874 00:23
163 1374.159058 1626.274414 1614.815918 11.381120 0.040864 0.735532 -0.682818 4987.481934 0.928477 1745.143433 1453.199219 0.487344 00:24
164 1373.824951 1683.784668 1672.551147 11.080626 0.036158 0.740005 -0.669310 5131.396484 0.927738 1745.729492 1441.605347 0.482697 00:23
165 1371.072754 1615.743530 1604.103882 11.577813 0.041652 0.736403 -0.681210 4918.139648 0.931138 1747.698853 1460.021118 0.490530 00:24
166 1355.484497 1641.220337 1630.196533 10.931018 0.036010 0.728178 -0.700613 5175.681641 0.913081 1720.974487 1422.012451 0.479188 00:24
167 1359.108276 1704.549927 1693.549438 10.845896 0.037629 0.718955 -0.726589 5429.520508 0.905963 1707.895020 1420.713013 0.481644 00:23
168 1356.763306 1649.856689 1638.938843 10.599342 0.041183 0.716552 -0.732771 5240.663086 0.901213 1699.459473 1412.924438 0.479067 00:24
169 1349.421021 1664.265259 1652.907837 11.272820 0.044008 0.724586 -0.713310 5020.294434 0.916043 1721.019531 1441.487793 0.488975 00:24
170 1350.994751 1623.858765 1612.932129 10.863392 0.041908 0.722831 -0.716796 5034.023926 0.908996 1710.285278 1424.442871 0.484604 00:24
171 1351.224976 1680.315918 1669.719727 10.547542 0.039740 0.710513 -0.750267 5435.653809 0.893453 1683.916748 1405.795776 0.481605 00:24
172 1346.770630 1604.967896 1593.335571 11.680385 0.039912 0.727918 -0.705782 4868.971191 0.924622 1739.492676 1460.044800 0.494222 00:24
173 1342.060303 1586.324097 1575.160645 11.029417 0.039247 0.720196 -0.726324 4858.954102 0.910087 1709.755005 1435.786011 0.491581 00:24
174 1338.160156 1588.046753 1577.109497 10.771853 0.035214 0.718428 -0.730816 4868.910156 0.906129 1701.200806 1427.917358 0.490517 00:24
175 1332.239746 1641.543823 1629.458740 11.952484 0.044998 0.719480 -0.730327 4942.689453 0.922858 1739.966675 1472.141724 0.498147 00:24
176 1325.793579 1655.992920 1643.510742 12.296820 0.043069 0.721262 -0.726518 4958.390625 0.929458 1751.547485 1487.265381 0.502298 00:24
177 1320.107178 1602.318115 1592.141968 9.982615 0.040755 0.695351 -0.794835 4962.369629 0.875475 1646.546143 1393.858398 0.486541 00:24
178 1316.732788 1599.840820 1589.633057 10.056231 0.039106 0.693247 -0.800047 5112.582520 0.871944 1642.347046 1387.680298 0.483511 00:24
179 1316.269165 1575.963501 1565.661133 10.261293 0.040414 0.692616 -0.804281 4894.453125 0.875071 1645.201904 1401.043335 0.491689 00:24
180 1312.937378 1565.147705 1554.971313 10.095403 0.039648 0.688514 -0.816473 4870.932129 0.870049 1635.080322 1397.676392 0.492972 00:23
181 1310.258423 1560.136353 1550.039917 10.130189 0.040148 0.689681 -0.813038 4840.253906 0.870697 1635.572021 1396.965210 0.492491 00:24
182 1303.114990 1559.687256 1549.733643 9.865709 0.039047 0.681105 -0.837071 4862.216797 0.858795 1614.653198 1384.084106 0.489868 00:24
183 1298.939209 1550.950439 1541.101929 9.818651 0.037294 0.678011 -0.846824 4839.537109 0.855971 1607.928955 1384.890259 0.491908 00:24
184 1296.500000 1558.706543 1548.998169 9.687429 0.041066 0.672073 -0.864892 4854.323242 0.849401 1596.333740 1383.215210 0.493486 00:24
185 1293.234863 1547.689209 1537.501587 10.014702 0.038959 0.676473 -0.853218 4749.744141 0.860303 1615.902466 1401.379639 0.498134 00:24
186 1292.811646 1546.130127 1535.702148 10.226321 0.042530 0.678689 -0.847562 4762.799316 0.865040 1625.793945 1409.895508 0.501073 00:24
187 1290.489868 1535.805420 1526.125854 9.580198 0.038133 0.669057 -0.874888 4788.103027 0.846551 1589.678711 1384.710449 0.496846 00:23
188 1286.943726 1534.413452 1524.609619 9.683949 0.039481 0.668009 -0.878814 4777.341797 0.847518 1590.329102 1390.739990 0.499342 00:23
189 1284.725708 1535.779663 1525.948364 9.720008 0.038524 0.666964 -0.882472 4757.062012 0.847355 1589.900513 1393.358276 0.501523 00:24
190 1281.207520 1533.668091 1524.209351 9.349226 0.037252 0.661511 -0.897568 4794.526855 0.836779 1571.197021 1378.563721 0.496789 00:23
191 1281.659302 1533.689453 1523.680420 9.959739 0.038388 0.669413 -0.875679 4734.559082 0.852867 1600.474243 1401.654541 0.502732 00:24
192 1277.832642 1525.408081 1515.864014 9.418250 0.037457 0.659884 -0.903180 4768.744629 0.836689 1569.301270 1383.119507 0.499142 00:24
193 1280.964233 1526.267944 1516.641357 9.531398 0.038529 0.657949 -0.910114 4741.918457 0.836838 1569.322144 1389.669189 0.502961 00:24
194 1272.813477 1520.035156 1510.463379 9.607328 0.037974 0.659625 -0.904838 4727.374023 0.838028 1571.622803 1388.355347 0.502201 00:23
195 1269.634644 1519.734131 1510.097290 9.620123 0.039231 0.658118 -0.909451 4724.170410 0.836997 1570.078369 1389.426147 0.502542 00:23
196 1277.440552 1518.647949 1509.047485 9.579507 0.038892 0.658164 -0.909332 4728.376953 0.836522 1568.702026 1388.311279 0.502341 00:24
197 1281.807983 1518.112183 1508.432251 9.564679 0.039367 0.658109 -0.909897 4729.467285 0.837224 1569.640137 1390.796387 0.503976 00:24
198 1277.494141 1518.218018 1508.575928 9.431640 0.039109 0.655921 -0.916428 4730.639648 0.834341 1564.817627 1388.831421 0.503389 00:24
199 1279.022827 1517.873779 1508.196777 9.541930 0.039074 0.656656 -0.914409 4725.548828 0.835692 1566.833374 1390.649292 0.504218 00:24
{% endraw %} {% raw %}
prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')
Path('models/frozen-MMDVae-TMP-latent128-resblock-alpha20_2021-03-24_11.36.20.pth')
{% endraw %}

64 latents, alpha=10

{% raw %}
latent_dim = 64

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
     
    
    
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.33113112449646,
 0.05754399299621582,
 0.1943375587463379,
 0.13803842663764954)
{% endraw %} {% raw %}
#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mmd mu std logvar l1_b_mean mu_sd l1_latent_reg weighted_kld logvar_sd time
0 43863.750000 38415.714844 38403.582031 4.531198 -0.005897 0.299329 -2.509599 52373.148438 0.443983 215.868622 578.306458 0.593813 00:24
1 29051.519531 25556.123047 25545.445312 6.811570 -0.004425 0.374405 -2.064293 41032.382812 0.560199 277.017273 490.831116 0.606257 00:23
2 20931.611328 19844.193359 19834.179688 7.637002 -0.004937 0.397987 -1.930601 35639.593750 0.592585 290.151154 466.145905 0.562413 00:24
3 14663.121094 12426.390625 12416.396484 8.592687 -0.002675 0.421610 -1.805782 27062.740234 0.626879 308.991547 445.806946 0.527288 00:24
4 10193.139648 9417.468750 9406.019531 10.110537 -0.011966 0.455588 -1.648575 22221.794922 0.682492 335.222321 429.319397 0.516746 00:23
5 7524.279785 6364.255859 6352.415039 10.702779 0.011146 0.471224 -1.580377 17584.408203 0.701387 342.728180 421.015381 0.519270 00:23
6 6115.612305 5585.450684 5572.469727 11.632233 -0.000796 0.494548 -1.479531 15968.841797 0.733156 358.945038 410.607941 0.503936 00:23
7 5255.900879 5021.166016 5008.228516 11.662443 -0.007470 0.507035 -1.426110 14532.230469 0.745946 364.898315 403.718201 0.487873 00:23
8 4705.863281 4692.226562 4679.104004 11.865071 0.000847 0.522057 -1.364778 13638.712891 0.759972 370.586456 396.001831 0.477372 00:24
9 4339.275391 4509.605469 4495.935059 12.448554 -0.006662 0.537097 -1.307204 13147.251953 0.781810 380.401398 393.854126 0.474533 00:23
10 4063.217041 4339.301758 4325.839355 12.518625 -0.000959 0.539867 -1.295752 12419.667969 0.782707 379.506470 391.445526 0.469356 00:24
11 3844.226074 3915.171143 3901.546631 12.703267 -0.002186 0.557496 -1.232466 11357.519531 0.798980 386.335205 386.182617 0.474052 00:24
12 3663.054199 4350.450684 4336.080566 13.246813 -0.008105 0.564001 -1.208974 11926.161133 0.812054 393.216003 387.989990 0.474185 00:24
13 3521.080322 3717.507324 3703.758057 12.813422 -0.001015 0.569932 -1.187693 10647.582031 0.811043 391.481812 382.930420 0.473800 00:23
14 3417.034912 3798.777100 3783.877441 14.170590 -0.016727 0.591292 -1.114504 10922.300781 0.846533 408.519867 386.855652 0.475482 00:24
15 3295.059814 3525.010742 3510.413818 13.814376 -0.007868 0.580232 -1.151974 10026.403320 0.834055 402.132599 387.572205 0.472764 00:24
16 3211.870850 3535.393066 3520.083984 14.482000 -0.007838 0.606115 -1.062313 9636.927734 0.860866 415.887024 384.029480 0.462664 00:24
17 3143.329590 3317.885498 3303.194824 14.088883 -0.001076 0.601462 -1.077590 9092.801758 0.851771 410.015045 381.872803 0.463364 00:24
18 3076.614990 3438.985596 3424.058838 14.395399 0.000178 0.608181 -1.056857 9169.375000 0.863004 413.457825 384.296448 0.470794 00:24
19 3009.324219 3417.989014 3403.146240 14.122858 -0.007747 0.606345 -1.062125 9251.507812 0.858551 413.128448 382.520508 0.467891 00:23
20 2935.841309 3159.706299 3144.865479 14.208220 -0.003280 0.608992 -1.053701 8666.197266 0.862335 413.299194 383.190063 0.468349 00:23
21 2889.335449 3118.774902 3104.639648 13.635227 0.006028 0.614202 -1.034251 8542.797852 0.857498 410.562469 376.163757 0.459089 00:23
22 2904.316650 3182.259766 3166.236572 15.280206 -0.009078 0.650401 -0.921584 8301.858398 0.909851 435.520782 386.079926 0.463378 00:23
23 2842.148438 3260.038330 3245.093750 14.698691 -0.003224 0.629414 -0.985648 8522.114258 0.881078 422.249756 380.361420 0.460035 00:24
24 2805.152588 3302.236084 3285.666748 16.182764 -0.007153 0.645778 -0.935025 8513.980469 0.916546 439.027618 391.966736 0.460427 00:24
25 2785.724121 3376.423096 3359.880615 15.800976 -0.001768 0.657435 -0.901159 8544.414062 0.920507 438.538971 389.170227 0.466704 00:24
26 2762.451416 2964.311523 2949.358643 14.561893 -0.000443 0.647257 -0.929403 8000.426758 0.894137 427.106110 377.525421 0.458053 00:23
27 2724.360596 2976.501221 2960.346680 15.885023 -0.002973 0.651166 -0.919243 7846.898926 0.915923 437.352905 388.857391 0.463826 00:24
28 2703.101807 2907.522461 2892.239502 14.919702 -0.004438 0.654061 -0.908257 7749.744629 0.905100 431.247742 380.027710 0.455881 00:23
29 2670.739502 2896.980957 2881.499268 14.934103 -0.003205 0.652672 -0.913074 7585.269531 0.907845 433.303772 382.666962 0.456568 00:23
30 2657.177246 2908.962158 2893.283447 15.238147 0.002911 0.667215 -0.869191 7627.063477 0.921041 438.859680 382.833466 0.459951 00:24
31 2632.270264 2971.827881 2956.658691 14.708002 0.000174 0.651452 -0.916749 7723.069336 0.901899 430.017609 379.759186 0.459874 00:23
32 2633.249023 2896.732422 2881.563477 14.930392 0.000247 0.654663 -0.906587 7833.086426 0.906260 431.945465 380.294464 0.458233 00:23
33 2599.547119 2908.443604 2893.063965 14.918993 0.005476 0.661886 -0.884253 7560.501953 0.911457 433.857971 379.544250 0.454524 00:24
34 2592.372803 2866.823975 2851.997314 13.981042 0.007245 0.658070 -0.894802 7673.434082 0.896489 426.589722 372.266479 0.451590 00:24
35 2572.281494 2804.459961 2789.447754 14.607452 0.005032 0.661891 -0.885020 7414.482910 0.908774 433.066833 378.309479 0.458576 00:24
36 2559.514160 2804.730713 2789.944824 14.492640 0.009472 0.655917 -0.902291 7479.042969 0.900253 428.058014 376.270325 0.454680 00:24
37 2549.586182 2803.934570 2788.795410 14.567461 0.001700 0.673275 -0.849742 7306.768555 0.917193 435.740265 377.050690 0.453040 00:24
38 2542.187500 2838.276611 2822.072754 15.779968 0.002494 0.678474 -0.836513 7458.104004 0.938841 446.885040 388.480835 0.460749 00:23
39 2528.304443 2727.795166 2712.283447 15.043669 0.010385 0.680495 -0.828586 7119.888672 0.929208 441.239685 380.757141 0.454719 00:23
40 2509.762939 2745.618652 2730.799072 14.620611 0.006320 0.670531 -0.857315 7317.892578 0.913483 434.135284 375.863922 0.451071 00:24
41 2498.480957 2984.975830 2968.752930 15.645418 0.002487 0.689347 -0.803794 7466.937500 0.944765 448.014771 386.595825 0.455587 00:23
42 2526.743652 2784.272705 2767.412354 16.525219 0.008349 0.699563 -0.774746 7292.501465 0.962843 458.175140 393.207031 0.458788 00:24
43 2479.489502 2716.605225 2701.625977 14.440147 0.008998 0.675520 -0.843820 7103.064453 0.916906 435.091675 376.118988 0.454560 00:24
44 2450.944580 2820.361084 2805.691895 14.351919 -0.000299 0.681099 -0.825373 7175.885742 0.918579 435.982666 373.253693 0.447491 00:24
45 2399.970947 2855.157227 2840.262939 14.562766 -0.006594 0.668939 -0.862151 7546.999023 0.912756 435.205444 376.112946 0.453263 00:24
46 2318.381836 2948.972656 2933.513184 14.931308 -0.010820 0.669663 -0.858805 7707.319336 0.917600 438.631836 377.986450 0.446189 00:24
47 2233.905518 2531.709229 2517.131104 14.154142 -0.004406 0.668042 -0.863950 6903.244629 0.908768 432.607391 373.857697 0.447973 00:24
48 2210.590088 2912.146973 2897.474609 14.289447 0.000908 0.672408 -0.850297 7493.521973 0.910653 433.871857 372.607941 0.444153 00:24
49 2193.390869 2784.932617 2770.388428 14.446658 0.002993 0.675446 -0.841248 7174.827637 0.914766 435.479675 373.613892 0.444635 00:24
50 2167.483643 2572.924316 2558.306152 14.623973 -0.004008 0.672136 -0.852110 7034.622070 0.915774 435.759369 376.101990 0.448337 00:24
51 2148.242920 2437.702881 2422.453613 14.876254 0.001187 0.677917 -0.836370 6642.927734 0.924982 439.490601 379.451447 0.453160 00:23
52 2119.804688 2564.101807 2549.729980 14.138188 0.008252 0.682225 -0.822155 6762.032227 0.917301 434.697266 372.190369 0.446474 00:24
53 2106.597412 2382.819824 2368.285400 14.838985 0.008345 0.682144 -0.822582 6569.984863 0.921236 437.164886 374.518799 0.449668 00:23
54 2104.226807 2605.211182 2589.143311 15.944120 0.000012 0.704920 -0.759345 6805.437500 0.958070 453.829285 387.741425 0.458698 00:24
55 2091.646240 2432.240479 2417.528320 14.472643 0.014744 0.687805 -0.807240 6672.047363 0.926808 438.970825 375.887177 0.453741 00:24
56 2077.431396 2535.636475 2520.333496 15.055017 0.007748 0.706880 -0.751748 6741.846680 0.947529 448.772888 379.493866 0.449344 00:23
57 2097.115723 2535.762939 2520.549561 15.058545 0.010610 0.698099 -0.775957 6880.807129 0.943769 447.930084 380.545807 0.447572 00:24
58 2074.907715 2496.313721 2480.572754 15.564722 -0.000937 0.702782 -0.763651 6622.996582 0.953701 452.134033 385.031403 0.450153 00:24
59 2062.163086 2475.023193 2460.586914 14.161969 0.014412 0.689498 -0.801185 6833.764648 0.923114 436.862000 372.420868 0.448638 00:24
60 2049.748779 2374.473389 2359.209473 15.079268 0.011797 0.700543 -0.771383 6404.564941 0.945152 447.325958 381.644592 0.455207 00:24
61 2047.361084 2376.252441 2360.949707 15.071036 0.001268 0.705669 -0.756095 6325.954102 0.950458 449.741394 382.154999 0.452023 00:23
62 2030.971680 2297.718750 2283.258545 14.449381 0.006626 0.690856 -0.795822 6350.029785 0.926574 439.087372 372.843842 0.442442 00:23
63 2031.379517 2573.683350 2560.023438 13.541169 0.002363 0.680560 -0.826074 6887.925781 0.907360 429.354614 366.436218 0.443858 00:23
64 2022.141724 2335.422607 2320.044678 15.109181 0.009062 0.710958 -0.741824 6309.033691 0.955586 450.358704 383.545898 0.454912 00:24
65 2043.625610 2400.428955 2384.999512 15.008455 0.003341 0.713824 -0.731893 6451.890625 0.956545 453.298187 381.855011 0.446871 00:24
66 2024.132690 2468.580322 2452.721924 15.564773 0.003289 0.714655 -0.729227 6588.416992 0.962702 455.840393 384.967712 0.446715 00:24
67 1996.843994 2270.792725 2255.573242 14.917750 0.015514 0.715168 -0.729083 6179.713379 0.956190 451.436707 381.625122 0.453211 00:23
68 1996.723633 2653.511475 2638.159912 15.123527 0.009264 0.713429 -0.732334 6925.761719 0.956444 452.881958 381.582092 0.445393 00:24
69 1993.352417 2318.995850 2303.337158 15.534203 0.010102 0.723970 -0.703007 6181.556641 0.968995 457.881866 385.096649 0.445595 00:24
70 1997.632080 2365.313721 2351.059570 14.203370 0.013818 0.712918 -0.733379 6572.615723 0.942815 446.432953 373.483856 0.444729 00:24
71 1991.968140 2434.370850 2418.563477 15.534522 0.012001 0.728581 -0.691676 6287.305176 0.973917 460.384094 387.257904 0.450199 00:23
72 1984.693359 2430.349365 2415.005127 15.024178 0.005615 0.722411 -0.708394 6508.200684 0.964146 455.255981 383.346558 0.449675 00:24
73 1972.730835 2342.661621 2327.041504 15.494254 0.001184 0.722117 -0.709227 6325.833008 0.966910 457.598572 385.087433 0.448399 00:24
74 1981.359619 2544.892578 2528.841309 15.822600 0.009106 0.729326 -0.688972 6672.772949 0.978393 461.669189 389.258881 0.448416 00:23
75 1969.436523 2298.509766 2283.355957 15.099205 0.011240 0.722553 -0.707532 6332.187988 0.964057 455.776276 383.052948 0.446579 00:24
76 1951.999146 2364.954102 2349.126953 15.702997 0.008855 0.733929 -0.677247 6368.576660 0.980469 462.792969 389.344727 0.449776 00:24
77 1951.475342 2224.091797 2208.672607 15.312521 0.008498 0.733552 -0.676775 6025.102051 0.974773 460.427124 384.907806 0.446509 00:24
78 1957.876831 2426.103027 2411.799316 14.131270 0.007124 0.729588 -0.686136 6351.814941 0.955903 451.659302 374.227509 0.439011 00:23
79 1941.185791 2294.367676 2279.660645 14.518409 0.023894 0.723170 -0.703930 6132.000488 0.955547 452.413910 376.670837 0.441308 00:24
80 1943.570068 2236.377197 2221.678955 14.449833 0.007350 0.724631 -0.700530 6133.900391 0.957083 451.618469 377.185211 0.443405 00:24
81 1947.266846 2247.182373 2232.464355 14.676343 0.003813 0.726550 -0.694701 6146.789551 0.958115 452.276642 376.816711 0.441424 00:23
82 1957.837280 2534.509277 2520.347168 14.056922 0.016048 0.724160 -0.702193 6636.850098 0.950278 448.406555 373.571930 0.445898 00:24
83 1933.469727 2579.322266 2563.413086 15.722889 0.011245 0.737353 -0.666855 6630.823730 0.981821 463.296265 388.400604 0.447950 00:23
84 1942.276489 2363.916992 2348.039551 16.134193 0.012214 0.746446 -0.644253 6304.797852 0.994142 468.833160 393.955200 0.454514 00:24
85 1928.095215 2209.769043 2194.548340 15.230845 0.008082 0.734415 -0.675613 5984.231445 0.973779 459.524414 384.817932 0.450239 00:24
86 1921.092529 2302.175049 2287.004883 14.949891 0.009675 0.735890 -0.670195 6158.293457 0.971037 457.397339 381.883636 0.445640 00:24
87 1910.358398 2239.969727 2225.645508 14.069258 0.005961 0.724680 -0.700270 6194.545898 0.951774 448.981903 373.757874 0.443217 00:23
88 1922.070801 2302.062256 2286.641846 15.184399 0.019390 0.741756 -0.655016 6103.609863 0.977987 460.710144 384.518677 0.448399 00:24
89 1914.837036 2212.521240 2197.232910 15.218254 0.012381 0.749186 -0.635606 6004.154785 0.987106 464.512451 387.790405 0.451465 00:24
90 1908.945190 2239.150879 2222.606445 16.385725 0.010513 0.752060 -0.627548 6045.464355 1.004142 474.579285 397.498474 0.447366 00:23
91 1919.256104 2281.434326 2266.269043 15.264541 0.014341 0.755768 -0.616070 6124.239258 0.989729 466.477112 386.233459 0.442961 00:23
92 1900.793091 2262.195312 2245.846924 16.261921 0.009571 0.757675 -0.613236 5989.441406 1.008500 475.759094 398.756073 0.449456 00:23
93 1896.890991 2302.918213 2287.291748 15.300570 0.016387 0.750865 -0.629952 6144.000977 0.989973 466.276306 388.559601 0.445447 00:23
94 1889.807983 2357.636475 2342.659668 15.078780 0.010196 0.759209 -0.606814 6270.000488 0.991948 468.093231 386.461212 0.441518 00:24
95 1909.732300 2252.490234 2236.052490 16.184378 0.006094 0.758564 -0.610889 6073.866699 1.007757 474.724731 397.968323 0.451051 00:24
96 1888.111328 2201.737061 2186.147705 15.409701 0.005468 0.755066 -0.618677 5916.775391 0.993156 467.651886 389.073517 0.443881 00:24
97 1876.192383 2205.862305 2190.198975 15.518577 0.013440 0.755652 -0.618026 5960.430664 0.994487 467.427460 390.132782 0.448773 00:24
98 1891.700195 2235.016113 2218.613037 16.125742 0.010912 0.763022 -0.599383 5976.390137 1.011120 476.187958 398.826996 0.451252 00:23
99 1879.114868 2160.450928 2144.865234 15.580759 0.009719 0.757886 -0.611456 5865.650391 0.997542 469.790405 390.953796 0.446132 00:24
100 1878.698120 2264.475830 2250.002197 14.256703 0.019321 0.759118 -0.605667 6185.930664 0.982463 463.984253 379.972717 0.435502 00:23
101 1884.965210 2276.707764 2260.010742 16.751415 0.018522 0.774234 -0.570665 6064.729980 1.025500 482.417511 405.142883 0.452555 00:23
102 1868.320190 2195.005127 2179.216309 15.626679 0.018922 0.765999 -0.590748 5919.426270 1.005778 472.568909 394.234589 0.448182 00:24
103 1870.577271 2325.707764 2310.781250 14.914661 0.016117 0.764631 -0.593082 6181.680176 0.992466 467.353271 385.482513 0.442336 00:24
104 1861.645630 2164.401123 2149.490479 14.860444 0.010910 0.763384 -0.596666 5896.706543 0.992715 467.017334 385.989929 0.445966 00:24
105 1854.688232 2361.593750 2345.624756 15.899466 0.002891 0.776262 -0.563304 6310.805664 1.016886 478.773987 397.732208 0.443036 00:23
106 1858.104004 2166.645020 2151.011963 15.698688 0.003630 0.767692 -0.584930 5854.079102 1.005505 474.027863 392.670990 0.441261 00:23
107 1861.227051 2194.676758 2179.235107 15.386620 0.012783 0.762215 -0.599076 5973.272949 0.997356 469.537842 388.933960 0.442284 00:24
108 1853.402222 2214.352051 2197.413086 16.974899 0.008637 0.774503 -0.568865 5948.845703 1.028581 483.793304 406.355316 0.448923 00:24
109 1853.154053 2219.520508 2202.921387 16.433540 0.006969 0.773090 -0.571207 5958.150391 1.022447 482.378448 402.077209 0.442657 00:23
110 1844.206543 2221.476562 2206.081055 15.232481 0.017018 0.776417 -0.561965 5881.692871 1.007130 474.478088 390.759552 0.442425 00:24
111 1847.569580 2295.849365 2277.609619 18.011066 0.013199 0.793577 -0.520873 6022.988281 1.058052 498.998871 421.094086 0.450569 00:24
112 1844.456665 2218.931641 2203.596680 15.319036 0.012254 0.774684 -0.565667 6142.659180 1.008328 474.474670 391.739655 0.438111 00:23
113 1841.190552 2190.828613 2173.822998 16.907028 0.014819 0.792422 -0.522674 5820.849609 1.043826 491.324738 411.315399 0.446875 00:24
114 1835.547241 2351.720947 2336.396729 14.853375 0.018431 0.769720 -0.579932 6318.644043 1.000530 470.900970 389.046448 0.444679 00:24
115 1824.982056 2149.504150 2133.460205 16.047073 0.014818 0.784466 -0.541972 5829.170898 1.025810 483.199127 400.937317 0.443612 00:24
116 1829.949829 2174.595215 2159.585938 15.228032 0.007764 0.768903 -0.580901 6015.817383 1.000651 470.963165 388.486206 0.439754 00:24
117 1828.137207 2159.846191 2143.157959 16.303429 0.016402 0.774903 -0.568036 5800.884277 1.023074 480.765625 402.793121 0.450413 00:24
118 1821.919312 2172.395264 2157.207275 15.308952 0.022732 0.784529 -0.540608 5832.681641 1.012731 476.492767 391.950256 0.439437 00:23
119 1823.590820 2124.030029 2108.170166 16.110052 0.017425 0.787079 -0.534081 5774.553223 1.026570 483.151154 400.051453 0.439420 00:24
120 1816.651489 2134.261719 2118.056396 16.119879 0.017726 0.785489 -0.540190 5736.674805 1.027853 483.374786 402.598602 0.446004 00:24
121 1825.160400 2303.302979 2286.981445 16.150137 0.011127 0.790506 -0.526535 6017.223633 1.031576 485.434357 403.089417 0.442133 00:24
122 1812.840454 2192.228027 2177.015869 15.022361 0.017740 0.783207 -0.543806 5906.555176 1.012327 476.290436 391.977814 0.437572 00:24
123 1813.112915 2162.520996 2147.565430 15.085690 0.018595 0.787382 -0.533425 5929.552246 1.015461 477.697510 392.893707 0.439185 00:24
124 1811.771484 2091.887695 2075.936523 15.922107 0.012522 0.776765 -0.561986 5759.767578 1.015987 477.538086 397.007172 0.444426 00:24
125 1823.553345 2352.743408 2337.209717 15.474758 0.005156 0.791367 -0.523368 6327.042480 1.024082 481.934967 397.547821 0.436857 00:24
126 1805.034668 2180.511963 2164.601562 15.944650 0.011578 0.784811 -0.541834 5812.856934 1.026006 482.167419 401.474640 0.445950 00:24
127 1807.343872 2181.065918 2165.801270 15.445350 0.012048 0.786192 -0.537312 5858.233398 1.020407 479.858826 396.748749 0.442926 00:24
128 1801.989502 2138.415527 2122.885498 15.467233 0.016893 0.791176 -0.523486 5808.321777 1.023366 481.355225 396.787476 0.438674 00:24
129 1801.509277 2315.857666 2299.096436 16.754320 0.014647 0.804377 -0.493137 6029.151367 1.051027 493.211853 413.338501 0.448887 00:24
130 1797.859985 2107.056885 2091.309082 15.413078 0.011560 0.798054 -0.506407 5679.422852 1.030524 484.914032 399.735352 0.438550 00:24
131 1798.474731 2293.315430 2278.733154 14.416295 0.004536 0.777620 -0.555554 6227.265625 0.999133 472.173157 383.337585 0.428521 00:23
132 1803.351074 2255.859619 2239.011475 16.752981 0.010490 0.802543 -0.498135 5911.997070 1.048473 492.301392 412.187164 0.449776 00:23
133 1802.076660 2212.726318 2196.369629 16.381973 0.015475 0.795316 -0.514360 5925.979004 1.036741 486.914734 405.199432 0.443403 00:24
134 1778.235596 2241.857422 2226.428223 15.551047 0.016160 0.792991 -0.520058 5921.941406 1.024173 481.234497 397.460571 0.442906 00:24
135 1798.828857 2215.352783 2199.614746 15.520594 -0.000083 0.798767 -0.504301 5998.493164 1.031573 484.854431 399.885162 0.437686 00:24
136 1791.604004 2172.814941 2156.524902 16.056366 0.007524 0.799264 -0.503785 5884.136230 1.040192 489.181946 405.979462 0.440864 00:25
137 1795.246948 2234.702148 2219.749023 14.842214 0.014538 0.800434 -0.499434 6144.828613 1.022123 480.883423 393.029816 0.435652 00:23
138 1793.262695 2083.352051 2067.292236 15.950337 0.011878 0.798019 -0.507013 5692.207031 1.035376 486.541351 403.273041 0.440748 00:24
139 1783.274414 2149.961670 2133.359131 16.134953 0.010936 0.801305 -0.499190 5738.727539 1.042928 490.078369 407.561859 0.442594 00:24
140 1789.629883 2315.940430 2297.484863 18.563274 0.007493 0.817955 -0.460288 5939.103027 1.081770 507.795471 431.535492 0.449273 00:23
141 1791.627930 2207.731201 2191.982666 15.709380 0.013139 0.822607 -0.445180 5963.995117 1.051367 495.090454 407.621033 0.436277 00:23
142 1786.481079 2178.852295 2164.209961 14.598886 0.018185 0.781908 -0.545324 6043.077637 1.001952 473.065735 384.565674 0.433211 00:24
143 1772.505615 2088.655273 2072.413574 16.312189 0.009611 0.806352 -0.487083 5623.616211 1.048144 492.382812 410.063354 0.443864 00:23
144 1773.765015 2246.583984 2229.790283 16.737934 0.010199 0.801941 -0.498106 5920.999023 1.049527 492.969116 412.185272 0.444252 00:24
145 1781.379272 2127.945312 2112.522705 15.241892 0.014805 0.808196 -0.481105 5745.976074 1.035487 486.594574 400.525848 0.439721 00:23
146 1775.792358 2542.838867 2527.221924 15.221790 0.017093 0.815300 -0.461849 6359.957031 1.040990 490.206329 401.406860 0.432750 00:23
147 1774.002686 2196.975098 2178.918457 17.970129 0.008381 0.819415 -0.454260 5786.255859 1.077251 507.290497 426.548492 0.440215 00:24
148 1763.355225 2454.446777 2438.522461 15.798021 0.018277 0.814125 -0.465027 6367.140625 1.044590 492.299805 404.380280 0.431745 00:24
149 1763.413696 2331.704346 2316.855713 14.775893 0.017670 0.800866 -0.497773 6180.981934 1.020494 481.215607 391.676636 0.433188 00:23
150 1766.779297 2101.832275 2085.813721 15.988742 0.010703 0.805431 -0.485994 5637.121094 1.041236 490.133606 403.671326 0.431689 00:24
151 1756.943115 2089.755859 2074.065186 15.755790 0.013304 0.808495 -0.479667 5680.389160 1.041188 489.413513 403.752808 0.435701 00:23
152 1757.968384 2153.202393 2137.951660 15.111240 0.014387 0.802591 -0.493400 5782.798340 1.028324 484.404449 396.194733 0.431953 00:24
153 1761.133423 2126.274170 2108.299316 18.150089 0.015727 0.824693 -0.441610 5691.865234 1.081061 508.173676 428.076447 0.442108 00:23
154 1754.218750 2100.638672 2085.303955 15.198414 0.013400 0.813222 -0.467146 5764.148926 1.037852 488.758392 400.037048 0.431861 00:23
155 1752.608276 2185.967529 2169.387451 16.396458 0.010934 0.819536 -0.452934 5783.447266 1.059863 498.460938 413.928986 0.436640 00:23
156 1761.809570 2086.500244 2070.962891 15.445103 0.017514 0.817518 -0.457789 5658.406738 1.046739 492.070557 405.519073 0.438143 00:23
157 1744.872314 2104.696533 2087.964355 16.573210 0.009364 0.818238 -0.456819 5663.222168 1.060570 498.739502 415.193909 0.439059 00:24
158 1755.324707 2210.587646 2191.904297 18.642618 0.007834 0.829451 -0.430653 5912.790527 1.094097 515.484314 436.298737 0.444831 00:23
159 1750.736938 2276.606934 2261.609619 14.890740 0.013379 0.804139 -0.489871 6127.239258 1.025618 482.895538 394.194458 0.435201 00:23
160 1745.393188 2089.162354 2074.119873 15.043243 0.015280 0.801667 -0.495509 5638.349609 1.024409 481.983551 393.741150 0.431860 00:24
161 1736.090942 2147.017822 2129.102783 17.569410 0.008687 0.817546 -0.459487 5712.206543 1.073218 504.761566 424.593384 0.442993 00:23
162 1738.413940 2367.807373 2351.743164 16.021133 0.013200 0.809847 -0.476627 6164.467773 1.043733 490.112000 405.359436 0.437047 00:24
163 1730.273926 2095.576904 2079.865723 15.692489 0.015031 0.804401 -0.490447 5642.889648 1.037469 487.596954 402.597015 0.438198 00:23
164 1732.973877 2174.367676 2157.667725 16.617126 0.014107 0.807602 -0.483685 5860.258789 1.053869 495.876831 413.453308 0.443348 00:23
165 1735.284668 2110.953857 2095.302979 15.724959 0.004811 0.813329 -0.467449 5644.609375 1.044499 491.805328 404.473846 0.435122 00:23
166 1714.356567 2040.991577 2025.521973 15.638241 0.014943 0.812154 -0.470701 5513.529297 1.041535 489.991791 403.137115 0.435767 00:23
167 1712.174316 2073.326416 2059.069824 14.374624 0.011469 0.785766 -0.535316 5697.384277 1.003952 473.134125 384.395599 0.431473 00:24
168 1707.242554 2032.528687 2017.372681 15.232775 0.021898 0.799380 -0.502715 5552.247070 1.024798 482.139496 395.457214 0.437737 00:24
169 1701.608398 2100.842285 2086.958496 14.016817 0.014957 0.784930 -0.536719 5804.063477 0.995842 470.380188 379.255341 0.428535 00:24
170 1705.426025 2127.073975 2112.207520 14.765400 0.014636 0.791323 -0.521682 5771.546875 1.013267 477.264832 389.201874 0.432916 00:23
171 1697.164307 2029.953979 2015.340576 14.608382 0.009948 0.790753 -0.524450 5598.907227 1.012656 476.437225 389.680969 0.437537 00:24
172 1701.335815 2101.848389 2087.525146 14.297215 0.010021 0.777077 -0.558810 5761.908203 0.997443 469.637146 383.354065 0.436209 00:24
173 1687.971802 2035.941162 2021.396240 14.679951 0.011481 0.782743 -0.545734 5502.674316 1.005859 472.835815 387.943665 0.441890 00:24
174 1688.044556 2100.105225 2084.287354 15.559942 0.014354 0.788969 -0.529777 5633.252441 1.025336 482.666351 398.882202 0.441224 00:23
175 1676.035522 2010.885132 1996.957642 13.989811 0.008462 0.779063 -0.553532 5533.621094 0.994666 467.717865 380.985687 0.434618 00:23
176 1671.125122 2004.597290 1990.845215 13.638359 0.008967 0.768718 -0.580175 5492.801758 0.981971 461.985138 375.944366 0.435246 00:23
177 1665.241699 2052.365723 2037.499146 14.771413 0.009575 0.782816 -0.544561 5539.471191 1.009021 474.567261 389.439423 0.437188 00:24
178 1665.223999 2045.858154 2032.487183 13.229947 0.010469 0.767099 -0.583029 5713.245605 0.973069 458.953705 370.179657 0.429697 00:23
179 1666.685669 2031.299805 2017.070679 14.342589 0.009825 0.770655 -0.575405 5579.201172 0.990290 466.026184 380.720123 0.435823 00:24
180 1663.915894 2028.667969 2014.607666 13.977999 0.016078 0.769739 -0.577955 5492.001465 0.987753 464.716583 379.614777 0.436295 00:24
181 1655.057007 1997.265015 1982.822021 14.302199 0.011690 0.766108 -0.587889 5421.292480 0.988498 464.869781 381.413086 0.438475 00:23
182 1647.470703 2001.200928 1987.254395 13.935952 0.011662 0.765908 -0.588284 5481.558594 0.982021 461.706665 377.253204 0.438083 00:24
183 1644.264404 2038.258911 2025.146362 13.109586 0.014235 0.759005 -0.605171 5534.289551 0.964562 454.472351 367.980865 0.433600 00:24
184 1644.385620 2047.144287 2032.844604 14.502955 0.011979 0.762904 -0.596886 5548.737793 0.986703 463.879425 381.653381 0.439943 00:23
185 1636.942871 2006.265381 1991.897583 14.197330 0.010794 0.762206 -0.599666 5424.100586 0.985522 463.111511 381.617706 0.443979 00:24
186 1634.921997 1972.216187 1959.202271 13.357076 0.011709 0.746162 -0.640975 5457.113770 0.956162 449.686615 368.055725 0.439773 00:24
187 1629.582520 1984.731201 1971.263550 13.300425 0.012171 0.747330 -0.638034 5436.582031 0.958278 450.679535 369.046844 0.440862 00:24
188 1623.882446 1975.131104 1961.552612 13.839017 0.011932 0.753823 -0.620902 5386.224609 0.969848 455.789917 374.059143 0.440880 00:24
189 1627.823242 1960.349243 1947.042725 13.134436 0.012697 0.748932 -0.633633 5391.917480 0.959833 451.228424 369.377655 0.439953 00:23
190 1629.562866 1960.382568 1946.881348 13.574226 0.012456 0.750285 -0.630250 5368.029785 0.964561 453.250061 371.947876 0.440743 00:24
191 1618.520630 1960.600586 1947.427124 13.270995 0.012711 0.744906 -0.644753 5373.071777 0.957177 449.862152 369.332611 0.441584 00:24
192 1623.590454 1950.550171 1937.349487 13.391574 0.011325 0.741666 -0.653698 5364.424805 0.954702 448.554230 369.055206 0.442402 00:23
193 1614.169922 1949.909790 1936.643921 13.437939 0.011754 0.744202 -0.647078 5353.138672 0.957277 449.707184 369.869202 0.442982 00:23
194 1614.446899 1949.415161 1936.062012 13.407144 0.010088 0.741968 -0.652905 5357.693848 0.955644 449.010956 369.513000 0.442485 00:23
195 1610.449463 1948.377686 1935.371216 13.316976 0.011583 0.740351 -0.657302 5344.466797 0.951816 447.198792 367.799164 0.442478 00:23
196 1609.498535 1948.998901 1935.809692 12.996377 0.011523 0.733751 -0.675311 5363.760742 0.944965 444.183533 366.061371 0.443487 00:24
197 1607.115356 1946.807129 1933.686768 13.057825 0.011373 0.739433 -0.659943 5346.859375 0.952111 447.346741 368.376556 0.443361 00:23
198 1615.034180 1946.700073 1933.425415 13.417556 0.011178 0.741674 -0.653776 5341.712402 0.954577 448.507629 369.013855 0.442932 00:24
199 1613.031616 1946.369873 1933.160156 13.242634 0.010880 0.738820 -0.661524 5349.308594 0.951218 447.030548 368.003479 0.443090 00:24
{% endraw %} {% raw %}
prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')
Path('models/frozen-MMDVae-TMP-latent64-resblock-alpha10_2021-03-24_12.57.06.pth')
{% endraw %}

64 latents, alpha = 20

{% raw %}
latent_dim = 64

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
     
    
    
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.33113112449646,
 0.03981071710586548,
 0.18547092080116273,
 0.11481534689664841)
{% endraw %} {% raw %}
#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mmd mu std logvar l1_b_mean mu_sd l1_latent_reg weighted_kld logvar_sd time
0 40959.515625 37905.636719 37891.761719 8.208526 -0.024008 0.321359 -2.330088 51717.753906 0.548548 277.020477 558.442444 0.462210 00:23
1 28461.808594 31310.380859 31296.017578 11.071305 -0.001257 0.356824 -2.144540 45347.582031 0.634957 318.794250 540.526062 0.561912 00:24
2 20569.023438 24270.050781 24254.761719 12.098309 -0.021030 0.391993 -1.954344 39474.792969 0.675133 337.234344 506.135071 0.548326 00:23
3 14445.319336 15024.644531 15008.929688 13.079365 -0.020070 0.419365 -1.815197 30104.261719 0.710429 353.417725 484.618500 0.530492 00:24
4 10298.541016 9405.922852 9392.183594 12.310252 -0.012689 0.445155 -1.688505 22990.738281 0.714922 355.908539 453.056366 0.504430 00:24
5 7728.354980 6665.394531 6650.482422 13.403973 -0.013079 0.452304 -1.656670 18389.359375 0.737011 364.198914 455.659149 0.502500 00:23
6 6190.182129 5603.939941 5589.669434 12.933229 -0.013793 0.479955 -1.534696 15767.240234 0.748866 369.001404 430.846649 0.491857 00:23
7 5236.472656 5212.584961 5196.565430 14.607766 -0.016758 0.491669 -1.485687 14362.958984 0.785005 385.483612 437.185181 0.486615 00:23
8 4617.360840 4473.277344 4458.379395 14.115689 -0.030596 0.498461 -1.451891 12745.385742 0.778319 385.171143 424.746368 0.459398 00:23
9 4208.974609 4216.414551 4200.229004 15.131713 -0.027884 0.513088 -1.392592 11635.668945 0.803424 394.603180 423.435181 0.455078 00:23
10 3901.279785 4004.926025 3988.525146 15.441641 -0.022382 0.523603 -1.348659 10846.424805 0.816376 402.754547 419.191315 0.437780 00:24
11 3670.582275 3740.616455 3724.684570 15.171966 -0.032123 0.524357 -1.344947 10166.298828 0.812432 399.097137 416.266876 0.438142 00:24
12 3491.620117 3667.437744 3651.875488 14.801250 -0.026936 0.534644 -1.305375 9781.682617 0.815828 401.885956 408.844971 0.435637 00:24
13 3373.058594 3597.774658 3581.469238 15.714341 -0.029027 0.541989 -1.278454 9533.083984 0.833115 409.734467 412.325806 0.437646 00:23
14 3268.388672 3494.202637 3478.181396 15.434155 -0.029170 0.551247 -1.245216 9079.128906 0.836138 408.363190 406.691254 0.440574 00:24
15 3192.942139 3778.173584 3761.590820 15.698897 -0.017824 0.561030 -1.210492 9206.311523 0.848363 414.858185 405.788605 0.443565 00:24
16 3132.410400 3364.331055 3348.152100 15.786729 -0.019340 0.569641 -1.177485 8492.832031 0.853677 417.495789 401.234131 0.430003 00:23
17 3073.229492 3397.922852 3381.511230 15.826709 -0.029253 0.571797 -1.171749 8525.550781 0.855620 416.931549 401.589661 0.439019 00:24
18 3024.521240 3429.331299 3412.252441 16.369137 -0.021574 0.587442 -1.115198 8408.209961 0.877177 429.384705 401.066406 0.427119 00:23
19 2968.729004 3244.765137 3229.118896 15.019815 -0.023893 0.582480 -1.132304 8225.012695 0.854739 416.707214 391.953735 0.428889 00:23
20 2925.205078 3132.208984 3115.920166 16.067533 -0.026519 0.596821 -1.084884 7874.996582 0.875232 426.971710 394.178131 0.432377 00:24
21 2889.247314 3118.166748 3102.901123 15.002897 -0.029995 0.585165 -1.121761 7893.246582 0.851289 415.738831 387.821686 0.424242 00:24
22 2845.302734 3130.704346 3114.590576 15.593694 -0.020829 0.592054 -1.099054 7766.731934 0.867235 421.982574 392.017334 0.426566 00:24
23 2807.441162 3189.631836 3173.846924 14.906292 -0.021064 0.596690 -1.083462 7741.119629 0.862720 421.786804 386.648254 0.426360 00:23
24 2784.897949 3024.525879 3008.107178 15.926530 -0.020563 0.606408 -1.051424 7504.205566 0.881879 429.748260 391.035858 0.426201 00:23
25 2770.114746 3029.304199 3013.619873 15.341253 -0.023551 0.608746 -1.042673 7513.146973 0.875445 425.717957 385.517975 0.421918 00:24
26 2761.279541 3091.217041 3074.397949 16.858736 -0.025576 0.628417 -0.979442 7453.478027 0.908650 441.615448 392.550476 0.421370 00:24
27 2730.678223 2962.628418 2947.173584 15.276114 -0.026963 0.620887 -1.002515 7304.295410 0.881647 428.941559 381.008789 0.418781 00:24
28 2711.719971 2964.291016 2948.281006 15.699995 -0.030957 0.629126 -0.976302 7354.960449 0.894260 436.180389 383.461853 0.418827 00:24
29 2698.613770 3050.802002 3035.411377 14.809456 -0.014705 0.621483 -1.001865 7517.462891 0.878165 427.194489 378.995758 0.423432 00:23
30 2680.392822 2954.147461 2937.375244 16.296328 -0.016200 0.636249 -0.954343 7354.759277 0.910993 443.337250 388.900665 0.419872 00:23
31 2648.716309 3001.396973 2986.461182 14.623203 -0.042457 0.618944 -1.007994 7513.009766 0.869564 422.928467 375.671539 0.415266 00:23
32 2552.094238 2815.679688 2799.725830 15.741020 -0.021439 0.625477 -0.987615 7107.967773 0.892987 435.489410 384.390106 0.415763 00:23
33 2500.885742 2701.386475 2685.672852 15.191346 -0.037839 0.645597 -0.924251 6860.041016 0.902515 438.233551 378.975342 0.412627 00:24
34 2422.822266 2719.418457 2703.131348 16.222771 -0.024178 0.639103 -0.945698 6888.277344 0.908967 442.192963 386.675140 0.419654 00:23
35 2360.017090 2604.458252 2588.693359 15.577497 -0.019458 0.639788 -0.944168 6656.376465 0.902958 438.855408 382.952759 0.422265 00:23
36 2323.858398 2629.382080 2613.721191 15.710530 -0.040843 0.643459 -0.932621 6623.768066 0.904269 438.465942 382.072937 0.420569 00:23
37 2305.494873 2616.138916 2600.468262 15.508165 -0.032108 0.642783 -0.933116 6720.474121 0.901037 437.298370 379.680634 0.414019 00:23
38 2282.103271 2587.460205 2572.022217 15.235188 -0.030462 0.647844 -0.918168 6601.239258 0.902206 437.860565 377.771149 0.416694 00:24
39 2269.184570 2588.785400 2572.932373 15.794830 -0.031420 0.647192 -0.920478 6507.833008 0.909731 442.258911 382.598511 0.418556 00:23
40 2255.821289 2634.927979 2619.576904 15.215096 -0.030368 0.643526 -0.933151 6652.128418 0.897648 434.839813 378.182709 0.424244 00:24
41 2230.150879 2608.832275 2593.164307 15.503767 -0.035302 0.645093 -0.927296 6532.506348 0.902899 437.093842 379.957489 0.419940 00:24
42 2219.906982 2527.319580 2511.794189 15.334104 -0.043030 0.645172 -0.927873 6361.107910 0.902230 436.562714 380.134918 0.423762 00:23
43 2201.091797 2495.561523 2479.723389 15.423490 -0.031208 0.653627 -0.899724 6393.499512 0.911191 441.788147 379.411285 0.415641 00:23
44 2189.597656 2575.081055 2559.551270 15.148734 -0.042203 0.655386 -0.895617 6543.202637 0.906966 438.339722 376.976593 0.418705 00:23
45 2178.476562 2444.961182 2429.966797 15.099843 -0.040230 0.649281 -0.913414 6305.620117 0.898885 434.712189 375.223419 0.415874 00:24
46 2172.310547 2482.452881 2467.760498 14.447921 -0.044035 0.654752 -0.896872 6357.943848 0.897477 434.102753 371.511932 0.416756 00:24
47 2141.238037 2456.285889 2440.410400 15.681372 -0.042566 0.663778 -0.870181 6212.209473 0.919044 444.409149 379.744659 0.419992 00:23
48 2154.861328 2546.916992 2531.172363 15.450172 -0.033943 0.676151 -0.832013 6335.190918 0.928487 449.619537 378.242828 0.413183 00:24
49 2145.066895 2469.216064 2453.683838 15.684266 -0.025496 0.673677 -0.839913 6328.454102 0.927341 448.001160 378.854248 0.414005 00:24
50 2137.785156 2525.979736 2510.129395 15.720557 -0.032780 0.674190 -0.840757 6316.041504 0.928512 448.067810 380.668945 0.426630 00:23
51 2126.391846 2661.637207 2646.973389 14.391059 -0.040650 0.660632 -0.878130 6603.567871 0.901177 435.528046 370.034912 0.414009 00:23
52 2105.768066 2467.755615 2452.253662 15.424354 -0.037468 0.663489 -0.871305 6265.216797 0.917171 442.137299 378.603882 0.421058 00:24
53 2103.722168 2377.594238 2361.099121 16.275026 -0.033612 0.686561 -0.802462 6085.942871 0.949971 458.570190 386.814178 0.417211 00:23
54 2095.870850 2382.000000 2367.187012 14.688356 -0.040755 0.662407 -0.873287 6181.885742 0.906567 437.269135 372.609833 0.417340 00:23
55 2092.342041 2399.902832 2384.895508 15.042092 -0.039869 0.673240 -0.842401 6213.987793 0.919157 442.775543 375.259064 0.421195 00:23
56 2088.286133 2385.112061 2369.721924 15.278100 -0.046295 0.684825 -0.807242 6049.940918 0.930797 448.996765 376.266510 0.417435 00:23
57 2069.092773 2337.409180 2321.650635 15.621803 -0.043040 0.685400 -0.806601 5990.912109 0.937795 452.068726 380.653992 0.421472 00:24
58 2076.790039 2386.129395 2370.171387 15.846266 -0.048534 0.683600 -0.812022 6103.103027 0.938466 451.396606 382.240875 0.421836 00:24
59 2070.809326 2342.461426 2326.790039 15.471412 -0.040923 0.682762 -0.812756 6106.518555 0.934409 451.366760 378.775787 0.416978 00:24
60 2056.128906 2340.091797 2324.622803 15.560506 -0.040656 0.683655 -0.810514 6131.062988 0.931926 449.420929 377.251373 0.417531 00:24
61 2050.260010 2320.975098 2305.526855 15.350599 -0.048311 0.685417 -0.805873 6080.050781 0.931778 448.701385 377.091156 0.418551 00:24
62 2040.451538 2521.118408 2505.993408 14.938314 -0.049762 0.681335 -0.816348 6495.013184 0.922327 445.554321 372.675079 0.413930 00:24
63 2038.689331 2505.906250 2490.208496 15.594720 -0.043435 0.696972 -0.771599 6458.555664 0.945607 454.825287 379.383942 0.414439 00:24
64 2032.836426 2308.715332 2293.177246 15.184978 -0.040553 0.693830 -0.781028 6036.919434 0.939228 451.888916 376.933960 0.417453 00:23
65 2015.309570 2298.208740 2282.828613 15.287786 -0.036977 0.697089 -0.771004 5980.215820 0.942187 454.466736 376.905640 0.414263 00:24
66 2008.337524 2336.121826 2320.769287 15.371319 -0.043068 0.689534 -0.792985 6031.566406 0.934841 450.286255 376.155640 0.416149 00:24
67 2009.489014 2363.434326 2347.560303 15.824185 -0.039247 0.707292 -0.743578 5978.121094 0.956974 460.629425 382.485260 0.419388 00:23
68 2005.784668 2437.531982 2422.388184 15.217319 -0.043149 0.698698 -0.765850 6264.744629 0.939402 452.716766 374.706024 0.413257 00:24
69 2025.744385 2410.386963 2395.035645 15.130639 -0.049899 0.707194 -0.741309 6266.439941 0.944039 456.068146 373.874847 0.411469 00:24
70 2025.628052 2277.856445 2262.055420 15.804263 -0.043789 0.712316 -0.729428 5969.650879 0.960561 461.188538 382.785767 0.420261 00:23
71 1996.230591 2248.061035 2231.698975 16.225378 -0.043527 0.710262 -0.736371 5850.251465 0.965129 463.426025 386.929810 0.424471 00:23
72 1982.686035 2278.241943 2262.350586 15.812693 -0.040705 0.715838 -0.719557 5878.743652 0.963115 462.464539 382.576691 0.419891 00:24
73 1982.589844 2259.186768 2242.758545 16.206263 -0.045802 0.720160 -0.707639 5879.612793 0.972291 466.264282 386.838501 0.421261 00:23
74 1970.068115 2278.263428 2262.274414 15.968723 -0.040762 0.714737 -0.722584 5873.362305 0.962936 462.271088 382.977142 0.419814 00:24
75 1981.158447 2266.666016 2251.103516 15.459614 -0.040722 0.715182 -0.720492 5922.448242 0.956840 459.765747 378.735260 0.416658 00:24
76 1970.334106 2223.015625 2206.700928 16.313377 -0.042207 0.726375 -0.691620 5776.502441 0.978591 469.096100 388.800537 0.425368 00:24
77 1969.342285 2309.269287 2291.382324 17.535690 -0.040674 0.732879 -0.674238 5951.787109 1.001142 479.683197 400.779022 0.426885 00:24
78 1966.173462 2250.973389 2234.190918 16.659634 -0.042188 0.737850 -0.659704 5841.065430 0.992405 476.440338 393.024353 0.423177 00:23
79 1973.658325 2351.661133 2335.940186 15.611522 -0.048741 0.729052 -0.682376 6044.595215 0.969281 465.717041 381.239014 0.419188 00:24
80 1959.157715 2203.126709 2187.176270 15.913980 -0.046870 0.729033 -0.683844 5819.911621 0.973988 467.352020 384.998901 0.423520 00:24
81 1944.111694 2337.841553 2321.949707 15.870450 -0.046643 0.729944 -0.680425 5998.597168 0.973523 467.660278 383.818634 0.420256 00:24
82 1952.909058 2255.712158 2239.249023 16.176220 -0.042481 0.734449 -0.668277 5854.269043 0.983957 471.843292 388.691254 0.420209 00:24
83 1959.683838 2253.380615 2236.295410 17.104437 -0.046263 0.752321 -0.621303 5775.620117 1.008157 483.045502 398.637421 0.423044 00:23
84 1949.256958 2219.430420 2202.583740 16.743525 -0.054398 0.738995 -0.656493 5827.366699 0.993476 476.074493 394.050934 0.422816 00:24
85 1932.493774 2270.866211 2254.367188 16.498678 -0.048233 0.748392 -0.630448 5882.447754 0.998605 479.390198 393.025482 0.418640 00:23
86 1926.792236 2243.494141 2226.032471 17.444464 -0.044260 0.756213 -0.610334 5843.229492 1.016438 487.552429 402.216492 0.420767 00:24
87 1929.301270 2360.115479 2343.343750 16.728437 -0.040610 0.748089 -0.631461 6146.698242 0.999666 479.167206 393.764771 0.420261 00:24
88 1923.725220 2257.256104 2240.975586 16.156029 -0.049211 0.742427 -0.646089 5919.268555 0.986081 473.469788 387.036407 0.417948 00:23
89 1929.615967 2206.652344 2190.040771 16.590775 -0.036773 0.755979 -0.610598 5748.218262 1.006616 481.926392 395.595001 0.419618 00:23
90 1928.308350 2209.027100 2192.659668 16.364662 -0.044584 0.749799 -0.625688 5839.551758 0.997170 478.814575 391.007996 0.415506 00:24
91 1923.497681 2195.827637 2179.664551 16.160307 -0.047199 0.749959 -0.625790 5756.620605 0.993433 476.321106 389.024597 0.418284 00:24
92 1920.450562 2226.007080 2208.905762 16.849619 -0.048800 0.757970 -0.605939 5783.415039 1.012360 485.171051 399.320496 0.422795 00:23
93 1907.940796 2204.447998 2187.702637 16.519547 -0.051278 0.753737 -0.615771 5717.921387 1.000371 479.757446 392.281311 0.417495 00:23
94 1912.038574 2312.904541 2296.213623 16.322195 -0.055712 0.747593 -0.633354 5913.315918 0.995756 477.295959 392.347687 0.423129 00:23
95 1907.132446 2263.700195 2245.174316 18.527081 -0.038884 0.775146 -0.561486 5856.935547 1.044512 501.445770 414.818390 0.424095 00:24
96 1904.169800 2187.114990 2170.356689 16.821573 -0.050057 0.754459 -0.616189 5729.006348 1.008500 482.676453 398.399719 0.428970 00:23
97 1891.356934 2189.086426 2171.989014 16.857542 -0.047401 0.763935 -0.589764 5744.598633 1.013738 486.036896 397.815491 0.421385 00:24
98 1908.470215 2202.196045 2185.125488 16.816521 -0.056875 0.773643 -0.564541 5710.838867 1.024148 491.276215 402.244781 0.421891 00:23
99 1912.817383 2271.102783 2252.939941 18.390772 -0.055428 0.784456 -0.538195 5692.067383 1.047571 502.516022 415.232635 0.425635 00:24
100 1896.343994 2249.454346 2232.903809 16.592817 -0.043906 0.768404 -0.578345 5870.537109 1.015048 486.546722 397.331848 0.423663 00:24
101 1914.040161 2349.092529 2332.062744 17.087572 -0.054749 0.778082 -0.553636 5992.177246 1.027973 492.264893 403.656189 0.424343 00:24
102 1901.462280 2195.209717 2178.604248 16.544409 -0.051548 0.772895 -0.566246 5768.870117 1.018752 487.924347 398.592163 0.420112 00:24
103 1892.971924 2303.753662 2287.227783 16.524702 -0.060314 0.767669 -0.580470 5885.155762 1.012483 484.424347 396.762573 0.424053 00:23
104 1885.633057 2198.695068 2181.686523 16.892550 -0.054941 0.780200 -0.548623 5665.803223 1.028508 492.204071 403.483429 0.424624 00:24
105 1879.946777 2268.352051 2250.056885 18.080482 -0.048442 0.788029 -0.529151 5791.166016 1.048384 501.349243 414.562531 0.425883 00:24
106 1873.960815 2208.382080 2191.311768 16.957605 -0.047080 0.780603 -0.547422 5771.178711 1.028803 492.432068 403.063690 0.424380 00:24
107 1879.795288 2212.586670 2195.862793 16.702127 -0.053253 0.784210 -0.537922 5807.437012 1.027395 491.895325 401.629456 0.421658 00:24
108 1883.018921 2190.283203 2173.032227 17.055727 -0.051857 0.785534 -0.535363 5695.900879 1.035785 495.712860 406.933167 0.426740 00:23
109 1873.274658 2179.484863 2162.656982 16.483217 -0.055500 0.779132 -0.551298 5653.234863 1.023031 489.285736 400.304260 0.425169 00:23
110 1875.121094 2223.679199 2206.763184 17.019516 -0.049089 0.790508 -0.522450 5692.590332 1.038188 496.420776 406.762695 0.423776 00:24
111 1877.853149 2210.239258 2192.774658 17.271931 -0.052495 0.799981 -0.498773 5682.107422 1.049525 500.648621 412.238251 0.424388 00:24
112 1858.888428 2166.068604 2149.133301 16.933033 -0.045102 0.779267 -0.551682 5756.738770 1.029317 491.951874 404.509064 0.427517 00:24
113 1862.280884 2240.872559 2222.930176 17.940746 -0.050709 0.795631 -0.510101 5750.409180 1.053161 502.992889 415.951202 0.426617 00:24
114 1857.601562 2158.836670 2141.508301 17.422701 -0.062181 0.797177 -0.506004 5640.994629 1.047349 500.696991 412.163391 0.426705 00:24
115 1853.199829 2210.480225 2192.893066 17.271141 -0.047091 0.785792 -0.534970 5832.065430 1.039847 496.753998 409.455658 0.427171 00:23
116 1852.933838 2146.823730 2129.014404 17.883074 -0.053020 0.801176 -0.496569 5628.718750 1.056713 504.895203 417.359344 0.427777 00:23
117 1848.598389 2233.268311 2215.853027 17.289141 -0.059764 0.793226 -0.516765 5819.959961 1.042883 497.510315 410.493774 0.429645 00:24
118 1849.587036 2165.807861 2148.944824 16.706530 -0.054404 0.787493 -0.530290 5715.197266 1.031102 493.317596 403.317841 0.427419 00:23
119 1850.543701 2177.142090 2159.665527 17.332441 -0.047359 0.810525 -0.472348 5689.555664 1.059669 506.248169 416.212311 0.422900 00:23
120 1841.010986 2214.608643 2197.584229 17.009958 -0.057121 0.795853 -0.508748 5731.107422 1.040273 497.707611 407.221161 0.424813 00:24
121 1846.170898 2173.103516 2154.754395 18.223576 -0.047758 0.811087 -0.472756 5646.965332 1.069173 509.358124 423.510986 0.429530 00:23
122 1853.544312 2225.131348 2207.766846 17.305943 -0.046430 0.803354 -0.490419 5697.885254 1.052889 502.983093 413.446686 0.426195 00:23
123 1851.731934 2481.151611 2463.311523 17.763298 -0.055462 0.804994 -0.486807 6341.493164 1.058097 505.301697 417.067993 0.427814 00:23
124 1839.960571 2157.313965 2139.523438 17.576052 -0.047222 0.807782 -0.479942 5584.384766 1.058887 505.438995 416.808624 0.427958 00:24
125 1845.200439 2169.861084 2152.713135 16.921015 -0.054603 0.805359 -0.485298 5665.421875 1.050143 501.718750 411.613953 0.425129 00:24
126 1830.439575 2107.000488 2090.199463 16.751245 -0.053420 0.797460 -0.505693 5605.064453 1.041360 497.126251 407.748718 0.427595 00:23
127 1828.306396 2157.807373 2139.823242 17.962271 -0.043433 0.816684 -0.458812 5594.198730 1.073371 512.020569 424.879333 0.428176 00:24
128 1827.354248 2147.487549 2130.184570 16.906126 -0.050628 0.805090 -0.485073 5680.477539 1.050958 502.753296 411.344360 0.421612 00:23
129 1824.401855 2161.614502 2144.228760 17.148869 -0.054005 0.816206 -0.458426 5650.276855 1.062507 507.704712 417.061981 0.423682 00:24
130 1844.129272 2200.024414 2180.568848 19.540968 -0.045917 0.828700 -0.430714 5699.275391 1.100836 525.049561 442.097717 0.434318 00:24
131 1831.183228 2172.513916 2153.527588 18.783731 -0.059716 0.826740 -0.435194 5605.716309 1.090781 519.812622 435.964020 0.432458 00:24
132 1821.956543 2188.409424 2171.062012 17.084085 -0.045506 0.810349 -0.472832 5672.038086 1.056462 504.620270 413.819855 0.425505 00:24
133 1819.124512 2136.867676 2119.635498 16.931643 -0.055766 0.818459 -0.452607 5688.138672 1.060976 507.004974 415.650574 0.423561 00:23
134 1804.915527 2130.572754 2113.860840 16.581442 -0.046056 0.800802 -0.496614 5600.747559 1.041651 497.881439 406.528687 0.425530 00:24
135 1829.998047 2318.239502 2301.285645 17.017076 -0.055017 0.822902 -0.441829 5774.692383 1.064314 507.652069 416.891266 0.424127 00:24
136 1806.984863 2105.936279 2089.146729 16.699335 -0.052032 0.811204 -0.471221 5582.268066 1.053071 502.411713 412.099243 0.425724 00:23
137 1814.444946 2145.627441 2128.329834 17.153358 -0.060114 0.821226 -0.447187 5632.762207 1.064782 507.528839 418.476685 0.427403 00:24
138 1805.471313 2199.678223 2183.135742 16.372446 -0.052516 0.799314 -0.499490 5710.942383 1.037459 496.347778 404.100586 0.423365 00:23
139 1819.846069 2145.026855 2126.221680 18.958809 -0.051018 0.829187 -0.428853 5603.984375 1.090058 519.864563 434.282074 0.432143 00:23
140 1811.464722 2125.606445 2107.047119 18.525103 -0.053811 0.823253 -0.442563 5587.444336 1.083364 517.106812 430.668945 0.429754 00:24
141 1810.387451 2133.373535 2116.254639 16.819828 -0.051721 0.816452 -0.457239 5654.775879 1.057321 505.260498 413.226501 0.421894 00:23
142 1806.075317 2120.787598 2103.452393 17.282923 -0.052363 0.818737 -0.452253 5595.923340 1.063340 507.160065 417.151306 0.424902 00:23
143 1795.540649 2087.965088 2070.855713 17.126945 -0.056300 0.819899 -0.448234 5559.171387 1.061621 506.542542 415.230103 0.419480 00:24
144 1801.831665 2162.523926 2145.343262 17.049423 -0.057388 0.816673 -0.457555 5637.763184 1.059990 506.255005 415.492493 0.427217 00:23
145 1804.999756 2127.045654 2109.133057 17.737495 -0.055753 0.833211 -0.418142 5499.569336 1.082589 516.211426 427.903992 0.428208 00:24
146 1801.665161 2148.118164 2129.857422 18.056736 -0.052730 0.823558 -0.442566 5658.666016 1.079387 514.068542 428.214966 0.433466 00:24
147 1792.751709 2142.786621 2124.578369 17.868423 -0.052998 0.824698 -0.439490 5533.171875 1.079319 513.897949 427.638977 0.431516 00:23
148 1797.831421 2173.990479 2155.276611 18.601061 -0.054440 0.836886 -0.410921 5611.578125 1.096845 522.010376 438.024567 0.434779 00:24
149 1796.638428 2228.538330 2208.895996 19.546438 -0.051051 0.832049 -0.423230 5779.147461 1.103489 525.546753 443.900330 0.435701 00:24
150 1786.904053 2110.445557 2093.834229 16.688715 -0.053678 0.815785 -0.459071 5550.644531 1.054214 502.954102 411.350952 0.424670 00:24
151 1812.827026 2143.489014 2124.361816 18.786152 -0.044347 0.832040 -0.422074 5653.475586 1.095074 522.185120 436.957123 0.433390 00:24
152 1799.185059 2233.176758 2212.192871 20.709408 -0.051705 0.853283 -0.372682 5672.157715 1.134729 541.690063 462.382355 0.435817 00:24
153 1812.112915 2154.590332 2135.016357 19.187141 -0.057483 0.858340 -0.359738 5551.562988 1.122521 534.372314 452.520325 0.430530 00:24
154 1794.944458 2121.451172 2104.009521 17.503822 -0.045161 0.834485 -0.414604 5645.078125 1.080042 515.233337 425.328003 0.427388 00:24
155 1780.827148 2111.433105 2093.209229 18.122953 -0.050780 0.832785 -0.420899 5528.623535 1.088834 518.056152 433.061554 0.436324 00:24
156 1770.763428 2098.217529 2080.120605 18.108927 -0.054287 0.830613 -0.425537 5497.437988 1.083789 515.907837 429.865967 0.433293 00:24
157 1771.306030 2212.417969 2195.476807 16.899473 -0.056625 0.827637 -0.429593 5933.847656 1.065003 509.781036 416.075287 0.422057 00:23
158 1776.353882 2087.676514 2069.845703 17.885731 -0.063730 0.826433 -0.435985 5542.777344 1.077119 511.914581 426.821350 0.435371 00:24
159 1776.064087 2080.614502 2062.247559 18.272722 -0.062080 0.835431 -0.413815 5428.992676 1.089563 518.572021 433.277313 0.432645 00:24
160 1774.724731 2097.684326 2080.735107 17.021824 -0.059728 0.833517 -0.416339 5483.957031 1.072949 511.427704 420.878937 0.425410 00:23
161 1768.203125 2102.173828 2084.953613 17.338186 -0.057752 0.834072 -0.416002 5466.453613 1.077056 513.391663 424.134766 0.428677 00:24
162 1762.727417 2126.273682 2107.730225 18.194901 -0.053858 0.836063 -0.412737 5493.123047 1.091242 518.968262 434.447998 0.433172 00:23
163 1762.384277 2095.414551 2077.247559 17.796741 -0.057061 0.834609 -0.416704 5458.758301 1.087707 516.845642 432.514740 0.435471 00:24
164 1759.913452 2083.792236 2067.239990 16.672506 -0.059163 0.816292 -0.458592 5500.871582 1.055506 503.126526 412.833466 0.426726 00:24
165 1747.516113 2071.969727 2055.133545 16.573025 -0.056396 0.821127 -0.446755 5463.743164 1.058248 504.766235 413.619568 0.427426 00:24
166 1744.018311 2064.967773 2047.782104 17.212856 -0.063836 0.824432 -0.440080 5421.344238 1.067539 507.714264 420.367859 0.432141 00:24
167 1742.466064 2068.240234 2050.914795 17.414539 -0.045116 0.823396 -0.442528 5505.398926 1.069879 509.680603 421.229095 0.431992 00:24
168 1736.565674 2115.032227 2097.228516 17.884319 -0.051872 0.827917 -0.431772 5554.646484 1.078394 513.569763 426.601929 0.431754 00:24
169 1726.438110 2094.499023 2078.260498 16.040751 -0.056866 0.812977 -0.466193 5595.799805 1.043719 497.359436 405.318756 0.424593 00:23
170 1726.953613 2092.002930 2074.789795 17.238066 -0.054976 0.817428 -0.457181 5503.426270 1.061785 505.293701 417.498047 0.432004 00:24
171 1720.478394 2023.062378 2005.817383 17.148363 -0.055556 0.816739 -0.458925 5351.890137 1.061608 504.868469 417.550446 0.431573 00:24
172 1721.542236 2067.517090 2049.719971 17.945301 -0.052620 0.818362 -0.455606 5418.785156 1.070646 509.704895 423.574829 0.433836 00:24
173 1708.759155 2082.524658 2066.103760 16.491518 -0.051301 0.810217 -0.474385 5473.530273 1.047432 498.662964 409.068451 0.429199 00:24
174 1706.208130 2031.275269 2014.445068 16.777117 -0.048412 0.808730 -0.478308 5359.035645 1.050466 500.265717 411.317261 0.430561 00:23
175 1701.491455 2031.757935 2014.522217 17.132797 -0.053099 0.809011 -0.478601 5361.452637 1.057613 503.989349 416.828125 0.434030 00:23
176 1712.591064 2032.570923 2016.312134 16.346331 -0.051801 0.800762 -0.498216 5353.327148 1.038494 494.662140 405.462494 0.431776 00:24
177 1703.767334 2032.762573 2016.700439 16.100601 -0.045140 0.793146 -0.517556 5367.807129 1.027594 489.408844 400.055969 0.432699 00:23
178 1696.929565 2034.451416 2017.335815 16.935680 -0.047335 0.803647 -0.492378 5366.276367 1.049806 499.729431 412.863647 0.436997 00:24
179 1691.728638 2027.025391 2011.435181 15.510817 -0.047971 0.790223 -0.524526 5341.384277 1.019632 485.479065 395.643524 0.431086 00:24
180 1688.377319 1997.419189 1981.489258 15.739227 -0.051798 0.787794 -0.531329 5316.716309 1.022885 486.853210 398.793182 0.434040 00:23
181 1680.450562 2002.943970 1987.047607 16.222668 -0.046601 0.792041 -0.520841 5311.688477 1.029636 490.141815 402.002472 0.434734 00:24
182 1682.406006 2005.146606 1988.929565 16.306120 -0.047890 0.786824 -0.534678 5320.902344 1.027030 488.672089 402.100708 0.436856 00:24
183 1669.528320 2011.062500 1994.679565 16.292412 -0.048375 0.784463 -0.541157 5322.189453 1.027725 488.892303 403.459290 0.438037 00:23
184 1672.674072 2008.345215 1993.459595 14.990308 -0.047234 0.774762 -0.563847 5394.271484 1.000445 476.556335 387.380463 0.430257 00:23
185 1665.695679 1988.411011 1972.459351 15.768681 -0.048152 0.785689 -0.537397 5283.599121 1.019853 485.200226 397.609192 0.435943 00:24
186 1666.120972 1998.574951 1983.510864 15.154258 -0.047654 0.775752 -0.561783 5345.802246 1.002263 477.347717 388.512878 0.432290 00:23
187 1661.368896 1979.690796 1964.505737 15.134271 -0.043517 0.771362 -0.573920 5275.512695 1.000368 475.981323 388.767670 0.435638 00:24
188 1657.698120 1975.420654 1959.854492 15.459913 -0.048410 0.775859 -0.562318 5257.902344 1.007531 479.383728 392.259430 0.435072 00:24
189 1654.200195 1983.949341 1968.890381 15.158071 -0.044278 0.765555 -0.589420 5348.042480 0.995840 473.733246 387.885468 0.437123 00:24
190 1649.508423 1975.866943 1960.587036 15.073768 -0.049580 0.772895 -0.570389 5253.031250 1.002092 476.614929 389.933502 0.436859 00:24
191 1649.842163 1966.593628 1951.392212 15.041107 -0.044199 0.763613 -0.594720 5270.642090 0.994350 472.985687 387.650299 0.438145 00:24
192 1645.297119 1968.905640 1953.764526 15.033599 -0.044947 0.764035 -0.593256 5280.130371 0.992676 472.116638 386.332794 0.436240 00:24
193 1639.833862 1965.644043 1950.847656 14.685172 -0.045316 0.756681 -0.612864 5268.566406 0.985101 468.704620 384.004883 0.437959 00:23
194 1642.912231 1967.354126 1952.551025 14.900321 -0.046941 0.762611 -0.597354 5261.707520 0.990159 470.920685 385.475586 0.438247 00:24
195 1636.274780 1964.972656 1950.088257 14.669127 -0.045640 0.761229 -0.600855 5247.117676 0.989161 470.440002 385.180847 0.437545 00:23
196 1643.799194 1962.548706 1947.747681 14.790215 -0.045076 0.758820 -0.607549 5262.417969 0.987627 469.754883 385.108734 0.439199 00:23
197 1641.221191 1961.913452 1947.017822 14.887654 -0.044684 0.757657 -0.610499 5259.751465 0.985865 468.896118 384.299011 0.438880 00:24
198 1643.093994 1962.706299 1947.690918 14.864224 -0.044520 0.758440 -0.608379 5261.253906 0.986949 469.398987 384.708618 0.438392 00:24
199 1636.304199 1961.836548 1946.998779 15.015165 -0.045666 0.762721 -0.597067 5244.823242 0.991566 471.577148 386.287170 0.438087 00:24
{% endraw %} {% raw %}
prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')
Path('models/frozen-MMDVae-TMP-latent64-resblock-alpha10_2021-03-24_14.17.51.pth')
{% endraw %}

vanilla-resnet beta-VAE

{% raw %}
class ResBlockBVAE(BVAE):
    """
    simple VAE with a _probably_ pretrained encoder 
    """

    def __init__(self,enc_parts,hidden_dim=None, latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE):
       
        """
        inputs:  
            enc_arch (pre-cut / pretrained)
            enc_dim
            latent_dim
            hidden_dim
            im_size,out_range
        """
        enc_arch,enc_feats,name = enc_parts

        # encoder
        #  arch,cut = xresnet18(pretrained=True),-4
        #  enc_arch = list(arch.children())[:cut]
        
        BASE = im_size//2**5
        enc_dim = enc_feats * BASE**2  # 2**(3*3) * (im_size//32)**2 #(output of resneet) #12800

        self.encoder = build_AE_encoder(enc_arch,enc_dim=enc_dim, hidden_dim=hidden_dim, im_size=im_size)

        in_dim = enc_dim if hidden_dim is None else hidden_dim

        # VAE Bottleneck
        self.bn = VAELayer(in_dim,latent_dim)     

        #decoder
        self.decoder = build_ResBlockAE_decoder(hidden_dim=hidden_dim, latent_dim=latent_dim, im_size=im_size,out_range=out_range)

        store_attr('name,enc_dim, in_dim,hidden_dim,latent_dim,im_size,out_range') # do i need all these?




#  THESE ARE INHERITED..
#     def decode(self, z):    
#         z = self.decoder(z)
#         return z
    
#     def reparam(self, h):
#         return self.bn(h)

#     def encode(self, x):
#         h = self.encoder(x)
#         z, mu, logvar = self.reparam(h)
#         return z, mu, logvar

#     def forward(self, x):
#         z, mu, logvar = self.encode(x)
#         x_hat = self.decode(z)
#         latents = torch.stack([mu,logvar],dim=-1)
#         return x_hat, latents # assume dims are [batch,latent_dim,concat_dim]
        
        
        
{% endraw %}

128 latents, alpha=5

{% raw %}
latent_dim = 128
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.0001737800776027143,
 0.0012022644514217973,
 0.0006880222645122558,
 0.0004570883756969124)
{% endraw %} {% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )
<fastai.learner.Learner at 0x7f2e8e9203d0>
{% endraw %} {% raw %}
# the defaults are pretty good for now
n_epochs = 10

#learn.fit_one_cycle(freeze_epochs1,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs) #, lr=1e-4,pct_start=0.5)
learn.fit_one_cycle(n_epochs)#, lr_max= base_lr)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 64307.953125 58279.812500 58243.433594 -0.050396 0.446575 -1.720978 633.586060 0.047932 64056.988281 0.950300 0.219880 0.624841 00:24
1 52256.531250 47517.250000 47401.242188 -0.095860 0.402663 -2.017131 589.741516 0.185387 58108.828125 0.767923 0.194185 0.970138 00:23
2 34960.160156 31226.734375 30933.964844 -0.076051 0.269091 -2.800119 751.623657 0.385156 45276.707031 0.674353 0.099892 0.940919 00:24
3 24776.472656 20445.687500 20104.564453 -0.063708 0.384149 -2.067982 556.550415 0.607671 36356.878906 0.703798 0.140935 0.863217 00:23
4 17783.804688 16285.233398 15865.710938 -0.067235 0.429491 -1.891830 517.371826 0.808860 31224.277344 0.709278 0.170626 0.993864 00:24
5 13278.303711 11126.129883 10704.387695 -0.060666 0.502716 -1.599754 443.023224 0.948876 24944.644531 0.696404 0.200470 1.058257 00:24
6 10607.653320 9407.301758 8986.120117 -0.058500 0.538358 -1.487504 420.064575 0.999986 22220.373047 0.692390 0.222120 1.125344 00:24
7 9076.883789 8579.513672 8182.976074 -0.048066 0.568540 -1.401409 394.949036 1.000000 20782.205078 0.669947 0.237349 1.191771 00:24
8 8298.398438 8123.069824 7729.753418 -0.048600 0.572593 -1.406226 391.598938 1.000000 20188.673828 0.652731 0.242710 1.237344 00:24
9 7958.245605 8007.626953 7610.983887 -0.048821 0.575821 -1.404351 394.969757 1.000000 19997.000000 0.657966 0.246665 1.262074 00:24
{% endraw %}

This initial "burning in" of the KLD regularization is very unstable...

{% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()
<fastai.learner.Learner at 0x7f2e8e9203d0>
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(6.309573450380412e-08,
 1.5848931980144698e-06,
 8.239944662591369e-07,
 3.162276129842212e-07)
{% endraw %} {% raw %}
base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final=1000)#,lr=1e-4)
learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 7446.696289 7002.534668 6594.965820 -0.056031 0.609424 -1.249221 406.261200 1.000000 16749.544922 0.762242 0.250232 1.164957 00:24
1 6559.509766 6201.054688 5791.714844 -0.041461 0.618311 -1.230625 408.183350 1.000000 14291.185547 0.769989 0.256279 1.197002 00:24
2 5965.905762 5736.044434 5315.900879 -0.065841 0.617567 -1.259662 419.067261 1.000000 13115.260742 0.771636 0.256352 1.277502 00:24
3 5546.066406 5484.937012 5098.352539 -0.031883 0.655282 -1.149140 385.752136 1.000000 12599.239258 0.741821 0.269175 1.304524 00:23
4 5216.171387 5357.946289 4967.701172 -0.048715 0.659577 -1.167782 389.449951 1.000000 11821.793945 0.728742 0.278477 1.384185 00:24
5 4968.410156 4974.411133 4575.067383 -0.037054 0.655091 -1.190241 398.651703 1.000000 11053.794922 0.738295 0.278551 1.407455 00:24
6 4758.382812 4930.711914 4530.911133 -0.042303 0.655234 -1.206222 399.300415 1.000000 10765.223633 0.727668 0.281477 1.447791 00:24
7 4598.325684 4773.492188 4384.264648 -0.028840 0.662661 -1.158184 388.670288 1.000000 10158.075195 0.732503 0.278200 1.385599 00:24
8 4481.006348 4676.386719 4256.562988 -0.047947 0.652907 -1.256124 419.240143 1.000000 9864.458008 0.734679 0.288794 1.560645 00:24
9 4396.450195 4542.811035 4142.692871 -0.031679 0.664870 -1.194383 399.589661 1.000000 9653.433594 0.725160 0.288026 1.493910 00:23
10 4277.703125 4514.132324 4093.090332 -0.038448 0.659772 -1.263761 420.567749 1.000000 9540.075195 0.722800 0.298543 1.615734 00:24
11 4211.627441 4451.823730 4059.461670 -0.017897 0.679049 -1.174503 391.879242 1.000000 9462.656250 0.704454 0.300546 1.540585 00:23
12 4159.201660 4370.717285 3985.004150 -0.029835 0.686301 -1.132567 385.226257 1.000000 9333.199219 0.711258 0.301613 1.484894 00:24
13 4089.183594 4338.795898 3938.385986 -0.027388 0.671400 -1.209984 399.877716 1.000000 9139.356445 0.703476 0.301768 1.563450 00:23
14 4024.176514 4243.605957 3844.971924 -0.038013 0.681120 -1.182844 398.243713 1.000000 8863.547852 0.705809 0.307898 1.564476 00:24
15 3992.836426 4182.538574 3768.346436 -0.015892 0.671695 -1.240414 413.820831 1.000000 8815.115234 0.708553 0.310647 1.627596 00:24
16 3938.035645 4254.535156 3836.140381 -0.021022 0.672339 -1.260805 417.951782 1.000000 8692.565430 0.700489 0.314746 1.681814 00:24
17 3915.346924 4136.510254 3734.750488 -0.037996 0.684208 -1.182841 401.324951 1.000000 8577.833008 0.708467 0.312179 1.580640 00:24
18 3851.800781 4099.753418 3683.509521 -0.035467 0.673168 -1.264503 415.946136 1.000000 8637.051758 0.690829 0.317606 1.686912 00:24
19 3829.080811 4061.842773 3656.622803 -0.020944 0.678215 -1.224614 404.792664 1.000000 8405.791992 0.691823 0.315146 1.633282 00:24
20 3814.291260 4050.546143 3625.365479 -0.017918 0.671568 -1.263860 424.740845 1.000000 8314.105469 0.712885 0.319519 1.662932 00:24
21 3790.595947 4220.412598 3789.602051 -0.022943 0.678726 -1.280803 430.250519 1.000000 8668.383789 0.700995 0.329524 1.751208 00:23
22 3769.847412 4021.063232 3616.341309 -0.031142 0.680278 -1.221621 404.374634 1.000000 8211.235352 0.687985 0.318570 1.633209 00:24
23 3756.498779 3959.629883 3536.583008 -0.020466 0.670928 -1.292554 422.621613 1.000000 8118.128418 0.686245 0.323532 1.728216 00:24
24 3706.817871 4126.592773 3708.841553 -0.023985 0.678128 -1.259123 417.260498 1.000000 8252.971680 0.689504 0.326434 1.692541 00:24
25 3687.163330 4039.027832 3624.490723 -0.023506 0.679744 -1.252728 414.078033 1.000000 8196.382812 0.684681 0.328139 1.685571 00:24
26 3664.687012 3962.185303 3553.306396 -0.020275 0.681983 -1.241518 408.559418 1.000000 8060.037109 0.678305 0.328026 1.677929 00:24
27 3655.794189 3935.266602 3520.048096 -0.026080 0.684890 -1.246303 414.766052 1.000000 8066.496094 0.683167 0.332905 1.703669 00:24
28 3633.048096 3900.002441 3477.207520 -0.031444 0.678460 -1.280428 422.386871 1.000000 7979.427734 0.682424 0.332304 1.739592 00:23
29 3605.916504 3902.634277 3490.624512 -0.027529 0.688832 -1.218063 411.608704 1.000000 7930.282715 0.693239 0.331436 1.667827 00:23
30 3597.799072 3864.459473 3440.049072 -0.031359 0.680125 -1.279481 424.118408 1.000000 7889.754883 0.684454 0.334076 1.745801 00:23
31 3577.369385 3835.365479 3421.598145 -0.031690 0.687918 -1.243465 413.432159 1.000000 7764.869629 0.677153 0.336913 1.711155 00:23
32 3567.163574 3808.568115 3392.390869 -0.030360 0.685564 -1.245607 415.832184 1.000000 7823.952148 0.684545 0.334472 1.705133 00:24
33 3556.911133 3839.799316 3426.487305 -0.025034 0.679174 -1.259462 412.899506 1.000000 8020.717285 0.675566 0.331511 1.691069 00:23
34 3547.733643 3787.851562 3366.338867 -0.033549 0.684411 -1.252932 421.170959 1.000000 7632.813965 0.692167 0.335533 1.710420 00:23
35 3498.793457 3750.230957 3341.822998 -0.024690 0.690246 -1.227408 408.039459 1.000000 7705.300781 0.674841 0.336464 1.692907 00:24
36 3497.622314 3779.739746 3356.960205 -0.031387 0.678122 -1.298752 422.325592 1.000000 7688.250488 0.666296 0.338358 1.765067 00:23
37 3518.062500 3770.506348 3337.693848 -0.026667 0.684909 -1.278552 432.439392 1.000000 7656.469727 0.695298 0.342245 1.764647 00:24
38 3493.117676 3758.422607 3342.844238 -0.021858 0.688443 -1.241507 415.205994 1.000000 7605.489746 0.681559 0.339791 1.703853 00:24
39 3465.591064 3732.348145 3307.750488 -0.019985 0.681497 -1.268380 424.199097 1.000000 7533.059082 0.689752 0.338435 1.715226 00:24
40 3459.664551 3786.777588 3376.685059 -0.013549 0.681292 -1.238933 409.652313 1.000000 7627.532227 0.682110 0.330924 1.658110 00:23
41 3466.934326 3731.796387 3311.013428 -0.029070 0.678285 -1.272447 420.435974 1.000000 7626.694824 0.682827 0.335078 1.704524 00:23
42 3442.351074 3738.564697 3302.626465 -0.041683 0.680691 -1.311230 435.543732 1.000000 7666.170898 0.680946 0.344484 1.805883 00:24
43 3429.302246 3697.198730 3261.440674 -0.023874 0.680301 -1.309720 435.364044 1.000000 7565.525879 0.682782 0.345298 1.794736 00:24
44 3421.857422 3716.597656 3288.889404 -0.018520 0.678248 -1.302927 427.315948 1.000000 7654.982910 0.672833 0.342392 1.766432 00:23
45 3432.586670 3763.864990 3343.723389 -0.013860 0.682215 -1.270576 419.691833 1.000000 7661.630859 0.677036 0.339797 1.728749 00:24
46 3409.428711 3733.609619 3305.357666 -0.026532 0.675712 -1.306996 427.861542 1.000000 7548.878906 0.673747 0.341761 1.756524 00:24
47 3396.132812 3671.088623 3244.154297 -0.027941 0.680358 -1.281522 426.605988 1.000000 7443.862793 0.685382 0.340985 1.736874 00:24
48 3388.988281 3633.996582 3204.664795 -0.028510 0.683743 -1.284135 428.964844 1.000000 7269.158203 0.683487 0.344635 1.762939 00:23
49 3374.890625 3655.472168 3228.736816 -0.025874 0.681976 -1.279506 426.387177 1.000000 7347.823730 0.683649 0.342960 1.738811 00:24
50 3374.364014 3661.433594 3226.835205 -0.027927 0.680691 -1.301747 434.195587 1.000000 7426.177734 0.685498 0.345380 1.777534 00:24
51 3377.024414 3646.464844 3221.161377 -0.040788 0.683618 -1.276678 424.889130 1.000000 7338.713379 0.679540 0.343833 1.744919 00:24
52 3367.656982 3639.472168 3208.727295 -0.026861 0.677835 -1.289724 430.378632 1.000000 7426.609863 0.690034 0.342502 1.726464 00:24
53 3353.731689 3598.407715 3163.959717 -0.032011 0.680891 -1.294131 434.070862 1.000000 7290.533203 0.690069 0.345623 1.758445 00:24
54 3350.532715 3618.042969 3183.317627 -0.024679 0.679061 -1.301106 434.440765 1.000000 7441.650879 0.688150 0.345228 1.761177 00:24
55 3328.891113 3591.654785 3151.659912 -0.024667 0.675619 -1.320078 439.684174 1.000000 7220.856445 0.689841 0.345016 1.782546 00:24
56 3320.435059 3628.756104 3184.838135 -0.023147 0.673282 -1.327310 443.469299 1.000000 7381.559082 0.695388 0.344838 1.779874 00:24
57 3319.204590 3589.929688 3161.705322 -0.029322 0.681353 -1.283165 427.894348 1.000000 7210.187988 0.684854 0.343528 1.745058 00:24
58 3333.072510 3605.556396 3159.020752 -0.022901 0.669782 -1.351565 446.191711 1.000000 7292.311523 0.687392 0.345130 1.809493 00:24
59 3301.860840 3583.256592 3147.702881 -0.018526 0.682041 -1.308480 435.160400 1.000000 7301.224609 0.678834 0.351069 1.792501 00:23
60 3289.162598 3571.066650 3126.065918 -0.025570 0.675508 -1.345270 444.585358 1.000000 7098.305664 0.679848 0.351047 1.829405 00:23
61 3285.661377 3577.617432 3150.016113 -0.022347 0.678402 -1.289324 427.211182 1.000000 7195.916504 0.681604 0.343973 1.729821 00:24
62 3292.790527 3545.149658 3106.085693 -0.020378 0.672821 -1.324593 438.766663 1.000000 7157.503418 0.687220 0.345001 1.767740 00:24
63 3273.931641 3576.871826 3127.184814 -0.030022 0.663642 -1.371861 449.397675 1.000000 7171.856934 0.685767 0.344284 1.808998 00:24
64 3286.415527 3542.644287 3110.177246 -0.025038 0.680133 -1.290051 432.100250 1.000000 7128.390137 0.689775 0.345855 1.743150 00:24
65 3295.292969 3554.997070 3110.141602 -0.031996 0.673540 -1.316412 444.468079 1.000000 7076.810547 0.704291 0.345098 1.755248 00:24
66 3287.177979 3522.089844 3080.620850 -0.039147 0.673940 -1.317083 441.162537 1.000000 7100.459961 0.695908 0.345708 1.758326 00:24
67 3241.864502 3488.717773 3024.696045 -0.025456 0.659230 -1.414685 463.702454 1.000000 7130.245605 0.688320 0.351045 1.855794 00:23
68 3192.238281 3457.201172 3000.528809 -0.025450 0.664398 -1.374820 456.514099 1.000000 6979.332031 0.696865 0.349181 1.810351 00:24
69 3159.333496 3475.097168 3000.846436 -0.023379 0.655233 -1.422044 474.188934 1.000000 7015.226074 0.712399 0.347041 1.856408 00:23
70 3117.958740 3506.858643 3045.154053 -0.011474 0.658447 -1.394947 461.412323 1.000000 7187.719727 0.700233 0.347915 1.812438 00:24
71 3089.015869 3462.144531 3013.021973 -0.024864 0.666162 -1.344185 448.933685 1.000000 7037.438965 0.701362 0.347054 1.756467 00:23
72 3063.737793 3324.433838 2869.878662 -0.035118 0.660016 -1.366300 454.371368 1.000000 6876.746582 0.703546 0.345587 1.760979 00:24
73 3027.214111 3331.492920 2870.182617 -0.011670 0.660568 -1.397654 461.116272 1.000000 6893.470215 0.693894 0.351798 1.828764 00:24
74 3017.817139 3321.054199 2865.548340 -0.025422 0.660219 -1.382914 455.403198 1.000000 6842.084473 0.691309 0.352129 1.784368 00:24
75 2992.729736 3256.467041 2794.733398 -0.019993 0.661402 -1.395585 461.590149 1.000000 6778.131348 0.693400 0.355003 1.819794 00:23
76 2983.455566 3289.244873 2831.101807 -0.029130 0.667984 -1.363975 457.947998 1.000000 6734.300781 0.701110 0.355252 1.800379 00:23
77 2976.864746 3239.218018 2781.242432 -0.028557 0.660307 -1.386050 457.947540 1.000000 6727.701172 0.694157 0.352464 1.793653 00:23
78 2956.301025 3304.631104 2842.858887 -0.027479 0.657182 -1.408286 461.591095 1.000000 6794.987793 0.687993 0.355320 1.808959 00:24
79 2945.924316 3334.847900 2866.822021 -0.024711 0.658704 -1.412751 467.761139 1.000000 6867.231934 0.696831 0.356441 1.834160 00:23
80 2924.367188 3198.714355 2731.728760 -0.018036 0.659534 -1.423019 466.900421 1.000000 6670.595703 0.685548 0.359238 1.858049 00:24
81 2906.989014 3215.365967 2753.775635 -0.021889 0.660180 -1.399861 461.448822 1.000000 6624.345215 0.690112 0.357600 1.810584 00:24
82 2896.910645 3215.521729 2746.866455 -0.023133 0.657917 -1.429965 468.538452 1.000000 6574.671387 0.685399 0.359657 1.859299 00:24
83 2868.951416 3202.712891 2736.927490 -0.030787 0.663033 -1.420170 465.648895 1.000000 6593.380371 0.678296 0.363546 1.870697 00:23
84 2858.278809 3162.275635 2696.020264 -0.019797 0.660603 -1.421584 466.094910 1.000000 6575.957520 0.682723 0.360896 1.858423 00:24
85 2846.207275 3103.525635 2629.209473 -0.024866 0.656847 -1.462483 474.175415 1.000000 6484.671875 0.671980 0.365991 1.908332 00:24
86 2817.036377 3078.863525 2616.457520 -0.022164 0.662941 -1.424628 462.297211 1.000000 6458.416992 0.666711 0.365627 1.872652 00:24
87 2798.375732 3076.187744 2600.413330 -0.032358 0.658778 -1.468475 475.640625 1.000000 6369.106445 0.666486 0.369961 1.928813 00:24
88 2771.174316 3070.136230 2606.374756 -0.024365 0.665567 -1.433004 463.692261 1.000000 6426.307617 0.658267 0.370472 1.902251 00:24
89 2762.876221 3072.244873 2603.819824 -0.026865 0.662920 -1.453525 468.355865 1.000000 6418.618652 0.655338 0.372392 1.920821 00:23
90 2738.615723 3011.625732 2538.651123 -0.022828 0.662585 -1.463570 472.897278 1.000000 6366.740723 0.658284 0.373876 1.937470 00:23
91 2722.517822 2995.657715 2518.491211 -0.018094 0.661352 -1.488024 477.079224 1.000000 6295.547852 0.649196 0.376828 1.975732 00:24
92 2708.709961 3014.197021 2543.201416 -0.019909 0.665528 -1.464587 470.887909 1.000000 6312.110840 0.647752 0.377463 1.954773 00:24
93 2697.568115 2966.013672 2492.254150 -0.020296 0.664736 -1.472951 473.693115 1.000000 6222.543945 0.648153 0.378263 1.965246 00:23
94 2692.429199 2959.528076 2484.598633 -0.019818 0.663212 -1.485151 474.830475 1.000000 6250.688965 0.642955 0.378584 1.979259 00:23
95 2679.370361 2948.991699 2474.185791 -0.017727 0.664275 -1.484554 474.726776 1.000000 6253.794922 0.641554 0.379589 1.984106 00:24
96 2667.490234 2942.266357 2468.315674 -0.019934 0.664913 -1.478830 473.874390 1.000000 6198.797852 0.643243 0.379516 1.976109 00:24
97 2663.017822 2939.738770 2463.631104 -0.018928 0.664092 -1.489275 476.029968 1.000000 6207.451660 0.640717 0.380436 1.990803 00:24
98 2661.197754 2935.853516 2459.528320 -0.019907 0.664481 -1.487758 476.260712 1.000000 6187.456543 0.641910 0.380615 1.990031 00:24
99 2662.050293 2935.732422 2458.348145 -0.019693 0.665127 -1.483711 477.305267 1.000000 6187.085449 0.647012 0.380451 1.986876 00:24
{% endraw %} {% raw %}
prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')
{% endraw %} {% raw %}
x = 1
{% endraw %}

128 latents, alpha=10

{% raw %}
latent_dim = 128
alpha = 10 # doubled because latent is half?
batchmean = True
useL1 = False



# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.00043651582673192023,
 0.002511886414140463,
 0.0014742011204361915,
 0.001047128695063293)
{% endraw %} {% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )
<fastai.learner.Learner at 0x7f2e8e940af0>
{% endraw %} {% raw %}
# # the defaults are pretty good for now
n_epochs = 10

learn.fit_one_cycle(n_epochs)#, lr_max= base_lr)
# #learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
# learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

# learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 57119.871094 48978.390625 48411.449219 -0.171113 0.467323 -1.606784 11681.308594 0.047932 57218.921875 4.143692 0.132895 0.589939 00:25
1 42990.976562 33256.042969 32888.765625 -0.111939 0.787082 -0.534524 1944.819458 0.185387 45751.316406 1.667504 0.184117 0.476903 00:25
2 27649.550781 18003.472656 17567.388672 -0.051287 0.614378 -1.090928 1118.868530 0.385156 32650.421875 1.105906 0.211503 0.686238 00:25
3 18592.869141 13246.469727 12702.623047 -0.040173 0.588392 -1.146373 889.270142 0.607671 27025.304688 0.926397 0.170838 0.591538 00:25
4 13824.781250 11302.725586 10674.322266 -0.025407 0.626357 -1.017098 773.015381 0.808860 24171.271484 0.872669 0.170972 0.585883 00:25
5 11246.221680 9732.048828 9055.437500 0.002667 0.623677 -1.060219 710.477600 0.948876 21821.728516 0.787523 0.195109 0.716930 00:25
6 9790.408203 9052.549805 8347.827148 0.037016 0.624793 -1.088983 702.386963 0.999986 20620.949219 0.755349 0.210199 0.831470 00:25
7 8893.800781 8546.672852 7901.442383 0.032866 0.649115 -1.026540 642.988525 1.000000 19937.876953 0.710113 0.222703 0.880089 00:25
8 8421.801758 8350.280273 7711.315918 0.032067 0.647427 -1.042604 636.768677 1.000000 19623.410156 0.692194 0.226795 0.915346 00:25
9 8228.472656 8303.927734 7656.038574 0.032340 0.642907 -1.063508 645.732849 1.000000 19511.349609 0.690796 0.228713 0.935445 00:25
{% endraw %} {% raw %}
# # the defaults are pretty good for now
# n_epochs = 10

# learn.fit_one_cycle(10)#, lr_max= base_lr)
# #learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
# #learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

# learn.show_results()
{% endraw %}

This initial "burning in" of the KLD regularization is very unstable...

{% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()
<fastai.learner.Learner at 0x7f2e8e940af0>
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(6.918309954926372e-05,
 1.4454397387453355e-05,
 4.181874846835854e-05,
 3.16227633447852e-05)
{% endraw %} {% raw %}
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final = 1000)#,lr=1e-4)
learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 7636.996094 7720.530273 7083.126953 0.019008 0.683685 -0.985189 635.105652 1.000000 17760.312500 0.687582 0.255679 1.082731 00:26
1 6885.542480 6894.079590 6297.856445 0.006627 0.707549 -0.936249 594.373535 1.000000 16222.292969 0.647742 0.266870 1.149313 00:25
2 6309.057617 6247.788574 5663.020508 0.013977 0.733179 -0.899275 583.153992 1.000000 14961.074219 0.627324 0.282289 1.256484 00:25
3 5862.837891 5824.015137 5272.883301 0.018640 0.760227 -0.806429 549.852478 1.000000 13915.848633 0.627636 0.282030 1.206542 00:25
4 5524.013672 5580.453613 4991.576660 0.000202 0.744082 -0.900245 587.568359 1.000000 13173.700195 0.615521 0.290766 1.348567 00:25
5 5247.176758 5352.265625 4772.904785 0.012758 0.746673 -0.887565 578.224121 1.000000 12680.643555 0.611426 0.288877 1.332583 00:25
6 5000.711914 5149.272949 4567.143555 0.021369 0.757468 -0.872349 581.102356 1.000000 12148.317383 0.609617 0.297950 1.367787 00:25
7 4822.857422 4957.025879 4346.383789 0.012121 0.743142 -0.944743 609.705933 1.000000 11592.447266 0.603416 0.300536 1.456099 00:25
8 4670.200195 4811.929199 4201.861328 0.023626 0.744356 -0.943792 609.286682 1.000000 11109.132812 0.600485 0.303331 1.457995 00:25
9 4552.351562 4701.762207 4124.187012 0.012166 0.752433 -0.885569 576.840942 1.000000 10700.002930 0.601379 0.294862 1.367969 00:25
10 4457.607422 4782.782715 4139.709961 0.030688 0.750395 -0.973056 642.217896 1.000000 10746.132812 0.605009 0.315268 1.573785 00:25
11 4389.464844 4600.456543 3989.061279 0.006628 0.754086 -0.906970 610.690735 1.000000 10181.538086 0.620296 0.304632 1.429988 00:25
12 4316.668945 4546.292480 3929.669189 0.008763 0.755007 -0.928994 615.957153 1.000000 10081.615234 0.603556 0.313128 1.485501 00:26
13 4249.416016 4501.651855 3910.774902 0.001233 0.753363 -0.902888 590.159973 1.000000 9931.499023 0.598335 0.304420 1.410765 00:25
14 4194.772461 4419.601562 3797.657227 0.011631 0.748259 -0.947026 621.403442 1.000000 9608.806641 0.604558 0.311978 1.482321 00:26
15 4155.236328 4390.953125 3742.381348 0.013369 0.749580 -0.965407 648.057068 1.000000 9297.745117 0.618429 0.318911 1.533816 00:25
16 4107.301758 4357.127441 3726.751709 0.014419 0.751795 -0.950445 629.798523 1.000000 9370.619141 0.605639 0.317125 1.514558 00:25
17 4080.357178 4326.577148 3693.467285 0.019456 0.757275 -0.930076 632.673401 1.000000 9206.707031 0.618530 0.317690 1.501742 00:25
18 4047.259766 4325.830566 3670.197754 0.016036 0.747400 -0.985562 655.035889 1.000000 9117.528320 0.611593 0.321978 1.569723 00:25
19 4017.182617 4268.327637 3646.175049 0.016499 0.751013 -0.935652 621.611816 1.000000 9072.065430 0.609426 0.314141 1.470098 00:25
20 3994.158936 4206.021484 3567.909668 0.020197 0.751552 -0.950140 637.651001 1.000000 8851.089844 0.615019 0.318669 1.509441 00:25
21 3978.268555 4256.660156 3621.489990 0.006194 0.745934 -0.969727 634.768066 1.000000 8992.529297 0.602167 0.318927 1.514271 00:26
22 3955.441650 4265.469238 3629.651123 -0.002801 0.752090 -0.953087 635.321167 1.000000 8820.404297 0.607925 0.321245 1.517332 00:26
23 3929.008545 4206.423828 3560.397217 0.014343 0.746978 -0.970275 645.613831 1.000000 8713.512695 0.614093 0.319624 1.526678 00:25
24 3908.563965 4153.493164 3523.141113 0.006290 0.751558 -0.946957 629.941040 1.000000 8658.349609 0.608749 0.317679 1.503032 00:25
25 3898.939697 4141.995605 3511.120850 0.015062 0.746144 -0.961228 630.455627 1.000000 8498.142578 0.604333 0.316788 1.497967 00:25
26 3885.578369 4103.011230 3454.336182 0.016320 0.742984 -0.985140 648.239014 1.000000 8472.931641 0.609011 0.321561 1.529650 00:25
27 3885.664307 4144.894043 3514.354492 0.007924 0.749983 -0.937831 630.100952 1.000000 8521.073242 0.619199 0.315660 1.465887 00:25
28 3863.140381 4171.204590 3528.229492 0.008750 0.749470 -0.969313 642.520569 1.000000 8679.597656 0.605489 0.324102 1.536831 00:25
29 3844.579834 4065.047852 3432.645752 0.014242 0.751757 -0.957529 631.998291 1.000000 8357.607422 0.599753 0.322359 1.528431 00:25
30 3806.262695 4071.528809 3425.390625 0.006845 0.749838 -0.974802 645.797546 1.000000 8272.665039 0.603490 0.326274 1.551809 00:25
31 3803.645508 4030.028564 3395.478516 0.018196 0.753908 -0.944004 634.221741 1.000000 8226.596680 0.611156 0.321838 1.508687 00:25
32 3798.153320 4027.861572 3400.078369 0.002770 0.757909 -0.921416 627.355713 1.000000 8197.963867 0.616403 0.321067 1.476825 00:25
33 3795.605469 4068.377686 3419.073730 -0.000919 0.747791 -0.983250 648.834473 1.000000 8331.038086 0.603142 0.326154 1.559857 00:26
34 3774.649170 4009.320557 3368.954346 0.006577 0.745278 -0.975523 640.058167 1.000000 8151.725098 0.603200 0.322254 1.523311 00:25
35 3797.206055 4070.147705 3423.396484 0.010856 0.747066 -0.977548 646.342896 1.000000 8184.187500 0.606127 0.324828 1.541001 00:25
36 3764.414795 3997.911377 3347.640381 0.000495 0.740897 -0.989360 649.856018 1.000000 8078.457031 0.609805 0.322489 1.524083 00:25
37 3738.304443 4000.742188 3355.457764 0.006559 0.746498 -0.979872 644.819946 1.000000 8064.453125 0.602224 0.326189 1.537351 00:25
38 3721.401367 4012.916016 3346.493408 0.014446 0.741339 -0.996954 665.997559 1.000000 7987.633789 0.622092 0.324676 1.547024 00:25
39 3733.728516 4002.178223 3343.464111 0.003242 0.744426 -0.993874 658.359314 1.000000 8037.950195 0.610395 0.326803 1.561654 00:25
40 3713.223877 3972.769287 3311.979980 0.027494 0.745946 -0.984094 660.406372 1.000000 7864.602051 0.618499 0.326929 1.545728 00:25
41 3725.438721 4000.221680 3349.538574 0.010210 0.749517 -0.965508 650.194214 1.000000 7884.955078 0.617907 0.324975 1.529040 00:25
42 3714.680908 4006.921143 3337.395020 0.006614 0.741850 -1.008360 669.165833 1.000000 7901.441406 0.614372 0.328160 1.577131 00:25
43 3684.118408 3953.133789 3283.846680 0.002745 0.745630 -1.007263 668.936707 1.000000 7932.915527 0.609230 0.331397 1.602322 00:25
44 3617.731934 3961.301758 3277.286377 0.008253 0.732352 -1.054439 683.772156 1.000000 8140.794434 0.605360 0.331876 1.616396 00:25
45 3527.310547 3771.926025 3082.391357 0.021190 0.732152 -1.045097 689.335327 1.000000 7549.060059 0.621409 0.328936 1.597990 00:25
46 3481.053223 3792.558594 3111.916016 0.014146 0.736934 -1.022333 680.446411 1.000000 7618.551758 0.623152 0.328771 1.574195 00:25
47 3449.490723 3723.544434 3044.811035 0.004227 0.736182 -1.023490 678.525024 1.000000 7445.786133 0.619757 0.330939 1.563343 00:26
48 3453.458984 3735.135986 3037.158936 0.002894 0.727846 -1.065144 697.809265 1.000000 7540.457520 0.619377 0.332484 1.605249 00:25
49 3440.742188 3698.924316 3023.513428 0.015087 0.739747 -1.005747 675.299194 1.000000 7419.930664 0.626200 0.329499 1.548569 00:25
50 3408.526855 3675.649902 3002.518066 0.007204 0.733653 -1.024227 672.936035 1.000000 7433.500000 0.616485 0.328433 1.547451 00:25
51 3414.460693 3745.192383 3054.504639 0.023048 0.737199 -1.023411 690.448914 1.000000 7412.087402 0.632781 0.331396 1.572160 00:25
52 3390.644287 3662.520020 2988.273438 0.007120 0.741236 -1.006877 674.001587 1.000000 7432.633301 0.620694 0.332220 1.557971 00:25
53 3382.684326 3674.317383 3003.882568 -0.000215 0.739698 -1.012987 670.337097 1.000000 7319.799805 0.612742 0.332873 1.558902 00:25
54 3380.493896 3745.933105 3057.020508 0.010614 0.740175 -1.024149 688.698853 1.000000 7467.467773 0.624081 0.336852 1.585891 00:25
55 3365.373047 3633.520752 2938.203613 0.000266 0.735362 -1.048191 695.175659 1.000000 7304.637695 0.618786 0.336548 1.616588 00:25
56 3360.692871 3634.707031 2946.971191 0.006276 0.732114 -1.034212 687.640747 1.000000 7217.326172 0.627182 0.330707 1.557482 00:25
57 3361.901367 3677.013184 2977.446289 0.008722 0.729319 -1.064693 699.587585 1.000000 7371.806152 0.619002 0.334934 1.612427 00:26
58 3341.074951 3619.520508 2932.792236 0.004525 0.736755 -1.030560 686.562866 1.000000 7233.354980 0.621963 0.333719 1.583878 00:25
59 3341.017334 3677.554688 2991.288574 -0.001353 0.736229 -1.028123 686.075989 1.000000 7211.670410 0.624420 0.332879 1.574305 00:25
60 3336.556885 3614.616699 2914.052734 0.012258 0.729867 -1.065539 700.479431 1.000000 7202.227539 0.618801 0.334756 1.621663 00:26
61 3338.885986 3666.255859 2975.747559 0.011388 0.735329 -1.025145 690.370422 1.000000 7230.252930 0.633762 0.331731 1.559722 00:25
62 3316.322510 3614.233643 2910.035645 0.011739 0.730760 -1.060256 703.997192 1.000000 7130.198242 0.625956 0.335665 1.610752 00:25
63 3306.802246 3591.011963 2898.936035 0.019578 0.729113 -1.051367 692.025085 1.000000 7151.365234 0.622282 0.331209 1.580528 00:25
64 3319.408691 3607.750244 2936.093506 0.015589 0.738210 -1.010526 671.561401 1.000000 7256.174805 0.619674 0.329484 1.550395 00:25
65 3329.283447 3666.196533 2997.034180 0.007939 0.738448 -1.005233 669.044373 1.000000 7284.672363 0.620451 0.329718 1.533679 00:25
66 3319.029053 3617.955322 2919.307373 0.009864 0.730675 -1.044238 698.430969 1.000000 7143.467285 0.634253 0.331163 1.574749 00:25
67 3304.009033 3641.074951 2941.493896 0.036581 0.734014 -1.039964 699.558655 1.000000 7150.225098 0.633657 0.332079 1.591718 00:25
68 3283.734131 3604.605469 2921.168701 0.009594 0.734629 -1.032366 683.271362 1.000000 7177.411621 0.619841 0.331710 1.576211 00:25
69 3282.202637 3547.863525 2858.945801 0.018959 0.732195 -1.035658 688.809265 1.000000 7019.934570 0.627719 0.329841 1.567864 00:26
70 3285.655762 3639.036133 2936.884277 0.013103 0.730776 -1.055586 701.959717 1.000000 7140.610840 0.627760 0.334509 1.601171 00:26
71 3288.370117 3591.168457 2893.040771 0.015652 0.726576 -1.061164 697.990662 1.000000 7123.018555 0.624936 0.331148 1.586430 00:25
72 3270.113281 3567.283447 2889.191162 0.009472 0.737333 -1.018427 677.971008 1.000000 7061.854004 0.621693 0.330927 1.562181 00:25
73 3268.113281 3598.905273 2899.352539 0.011853 0.725724 -1.065427 699.484924 1.000000 7144.051270 0.624485 0.331081 1.592555 00:25
74 3265.667236 3563.320068 2867.907227 0.012042 0.729498 -1.047444 695.306763 1.000000 7025.574219 0.628830 0.331620 1.572838 00:25
75 3245.814453 3517.445068 2836.110107 0.003315 0.734034 -1.036802 681.251587 1.000000 7037.501953 0.613656 0.333098 1.580311 00:25
76 3241.935303 3530.926758 2817.048340 0.011109 0.723914 -1.084425 713.788513 1.000000 6924.078125 0.627887 0.333674 1.624286 00:25
77 3245.354980 3546.132568 2842.161621 0.005974 0.725738 -1.074948 703.946472 1.000000 6994.406738 0.621140 0.333325 1.616471 00:25
78 3245.945557 3528.050781 2828.606201 0.012258 0.728167 -1.062054 699.372986 1.000000 6909.879883 0.623076 0.333196 1.599359 00:25
79 3229.143799 3506.233887 2806.143555 0.004439 0.724063 -1.080432 699.957153 1.000000 6967.483398 0.614117 0.332865 1.616866 00:25
80 3220.140869 3577.505371 2893.383057 0.005646 0.731227 -1.039378 683.952393 1.000000 6991.556152 0.620166 0.329861 1.570222 00:25
81 3208.395508 3488.423340 2788.742188 -0.000503 0.729512 -1.061916 699.554688 1.000000 6825.243652 0.620982 0.335146 1.605686 00:25
82 3191.760254 3466.438232 2781.985352 0.004644 0.736124 -1.030661 684.366333 1.000000 6905.499023 0.620341 0.332895 1.582907 00:25
83 3171.593750 3468.174316 2757.281982 0.013745 0.726387 -1.079011 710.753906 1.000000 6798.083984 0.624436 0.335499 1.625272 00:26
84 3161.748779 3484.867432 2804.370850 0.000396 0.730777 -1.043842 680.357971 1.000000 6938.908203 0.611897 0.331202 1.576850 00:25
85 3145.614746 3414.764160 2711.646484 0.007162 0.726800 -1.078855 703.033142 1.000000 6748.033203 0.614373 0.335700 1.630140 00:25
86 3135.015625 3464.562256 2775.013672 0.002118 0.727726 -1.071037 689.401733 1.000000 6989.595215 0.602924 0.334548 1.618078 00:26
87 3112.888184 3406.750488 2711.639893 0.013885 0.731807 -1.061808 695.017822 1.000000 6812.809570 0.611747 0.336112 1.625439 00:25
88 3104.485596 3372.153564 2665.243896 0.003058 0.727263 -1.091322 706.815735 1.000000 6685.791992 0.606084 0.340014 1.657788 00:25
89 3089.446289 3366.230469 2659.792236 0.007324 0.727721 -1.095376 706.328125 1.000000 6742.983887 0.601475 0.340201 1.673326 00:25
90 3073.190186 3344.774414 2630.106934 0.006065 0.724318 -1.111660 714.586670 1.000000 6634.008789 0.602436 0.340756 1.686546 00:25
91 3060.273926 3337.490479 2629.539795 0.006006 0.727197 -1.101563 707.863281 1.000000 6658.198730 0.598446 0.340994 1.685095 00:26
92 3047.108398 3325.574951 2611.993408 0.016005 0.727472 -1.106509 713.503357 1.000000 6600.878906 0.600146 0.342724 1.696152 00:26
93 3038.662354 3313.505615 2604.016357 0.005596 0.728518 -1.106411 709.403442 1.000000 6611.263672 0.593522 0.343257 1.704652 00:26
94 3017.475830 3303.448486 2587.911377 0.006252 0.727385 -1.113257 715.440002 1.000000 6552.413086 0.596630 0.343977 1.711420 00:25
95 3021.622803 3296.240723 2581.996338 0.005719 0.725646 -1.115477 714.178345 1.000000 6569.918457 0.595726 0.343124 1.704090 00:25
96 3009.111816 3291.238281 2580.051758 0.004037 0.726861 -1.111897 711.094604 1.000000 6562.848633 0.592960 0.343621 1.703392 00:25
97 3000.961182 3284.883545 2570.526123 0.008544 0.726318 -1.115299 714.283325 1.000000 6541.076172 0.594613 0.344068 1.706477 00:25
98 2997.922852 3285.489746 2576.586670 0.007900 0.727089 -1.110541 708.791443 1.000000 6581.392090 0.590692 0.343674 1.701313 00:25
99 2992.362305 3283.334229 2569.966553 0.007610 0.726215 -1.116118 713.273376 1.000000 6540.073730 0.592672 0.344181 1.707559 00:25
{% endraw %} {% raw %}
learn.lr
0.001
{% endraw %} {% raw %}
# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

# learn.save(filename)
# learn.export(f'{filename}.pkl')
{% endraw %} {% raw %}
# base_lr = 1e-5# gmlr #/= 2
# epochs = 50

# #learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
# #learn.fit_one_cycle(epochs, lr_max= 1e-3)
# #learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
# learn.fit_flat_cos(epochs)
# learn.show_results()
{% endraw %} {% raw %}
# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

# learn.save(filename)
# learn.export(f'{filename}.pkl')
{% endraw %} {% raw %}
# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"
# filename = "BVae-POST-1CYCLE10-latent128-resblock-alpha10_2021-03-24_21.33.02"
# learn.load(filename)
# #epochs = 5
# epochs = 10
# learn.fit_one_cycle(epochs, lr_max=.001)
# #learn.fit_flat_cos(epochs,lr=.0015,pct_start=.5,div_final=1000.0)
# #learn.fit_one_cycle(epochs,lr_max=5e-3,pct_start=0.5,div_final=100000) # gets down to ~4500 loss in 10
# learn.show_results()
{% endraw %} {% raw %}
prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')
{% endraw %}

64 latents, alpha=5

{% raw %}
latent_dim = 64
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False


# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.0002511886414140463,
 0.0030199517495930195,
 0.0016355701955035329,
 0.0008709632093086839)
{% endraw %} {% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )
<fastai.learner.Learner at 0x7f2e8f2804f0>
{% endraw %} {% raw %}
n_epochs = 10

learn.fit_one_cycle(n_epochs)#,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 62175.605469 58420.937500 58348.890625 -0.051949 0.756474 -0.705867 1368.726318 0.047932 61588.308594 1.822992 0.582341 0.659328 00:25
1 49618.242188 39473.527344 39206.046875 -0.043057 0.851662 -0.396860 1403.989502 0.185387 52113.386719 2.003324 0.221583 0.661778 00:25
2 38106.187500 29982.267578 29626.966797 -0.051806 0.686124 -1.142172 902.552917 0.385156 45277.328125 1.447544 0.294982 2.087242 00:25
3 31850.349609 25701.712891 25034.757812 -0.007050 0.893151 -0.651888 1084.826294 0.607671 40963.800781 1.650774 0.426548 1.889337 00:25
4 26937.095703 22336.453125 21880.179688 -0.020213 0.755051 -0.904856 554.679199 0.808860 37525.218750 1.066666 0.338944 1.718830 00:25
5 21971.119141 18084.140625 17574.322266 0.022983 0.661450 -1.064209 531.471375 0.948876 32476.261719 1.040300 0.269217 1.212421 00:25
6 17252.421875 14166.872070 13769.662109 -0.004297 0.650027 -1.045775 395.191437 0.999986 27589.769531 0.842583 0.236650 1.021509 00:25
7 13777.228516 12127.618164 11719.044922 -0.000522 0.590726 -1.253758 407.167450 1.000000 24565.738281 0.784295 0.232350 1.025303 00:25
8 12025.313477 11425.287109 11027.707031 -0.002325 0.577320 -1.306436 396.206085 1.000000 23534.054688 0.737082 0.233305 1.023820 00:25
9 11348.297852 11215.250977 10807.715820 -0.002401 0.565716 -1.354919 405.828369 1.000000 23167.085938 0.733784 0.232911 1.041644 00:25
{% endraw %}

This initial "burning in" of the KLD regularization is very unstable...

{% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()
<fastai.learner.Learner at 0x7f2e8f2804f0>
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(5.754399353463669e-07,
 2.2908675418875646e-06,
 1.4331537386169657e-06,
 1.1481544106572983e-06)
{% endraw %} {% raw %}
base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
#learn.fit_flat_cos(epochs,lr=1e-4)
learn.fit_flat_cos(epochs, div_final=1000.0)#,lr=1e-3)
learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 9465.326172 9999.078125 9601.225586 0.007130 0.609639 -1.205733 396.314056 1.000000 21853.699219 0.774112 0.246161 1.020246 00:26
1 8063.734375 7810.102051 7418.045410 -0.010892 0.615129 -1.217685 390.613068 1.000000 17732.515625 0.748079 0.253269 1.108232 00:25
2 7098.869141 6891.984375 6494.851074 0.003510 0.620044 -1.240355 395.655640 1.000000 15703.275391 0.735949 0.262632 1.223647 00:25
3 6485.855469 6270.389648 5868.511719 -0.005693 0.648472 -1.133566 400.792297 1.000000 13933.801758 0.791065 0.267172 1.181391 00:25
4 5972.832520 5844.192871 5451.658691 0.012890 0.644426 -1.175375 391.558289 1.000000 13166.174805 0.748259 0.268520 1.273546 00:25
5 5572.884766 5628.988281 5244.474121 0.007183 0.655677 -1.138401 383.767944 1.000000 12375.596680 0.746120 0.270278 1.270705 00:25
6 5250.625977 5428.687500 5025.041504 0.000618 0.657387 -1.176482 402.747681 1.000000 12075.447266 0.754705 0.281650 1.383355 00:25
7 4992.806152 5028.703125 4640.528320 0.009451 0.677483 -1.110796 387.616364 1.000000 11064.332031 0.747032 0.286410 1.379216 00:25
8 4797.666016 4977.150879 4576.464844 -0.004673 0.662541 -1.180220 399.917969 1.000000 10845.199219 0.740392 0.284689 1.444876 00:25
9 4619.204590 4714.991211 4327.709961 -0.004181 0.676296 -1.131910 386.741913 1.000000 10164.172852 0.729932 0.291930 1.414375 00:25
10 4479.482422 4887.600586 4481.559082 0.000063 0.672485 -1.165117 405.433441 1.000000 10460.187500 0.749355 0.294086 1.469483 00:25
11 4343.376953 4530.287598 4109.966309 0.000223 0.664114 -1.229507 419.752380 1.000000 9527.458008 0.741199 0.301180 1.557173 00:25
12 4236.599609 4525.162598 4090.072266 -0.011065 0.662348 -1.276214 434.732178 1.000000 9375.357422 0.739744 0.307770 1.653390 00:25
13 4161.453613 4408.749512 3994.785400 0.002932 0.679915 -1.200745 413.446655 1.000000 9209.034180 0.729387 0.309570 1.608523 00:26
14 4097.939453 4343.777832 3945.017334 -0.005056 0.684136 -1.152501 398.164246 1.000000 8986.760742 0.727446 0.305452 1.514880 00:25
15 4039.005859 4262.605957 3840.835449 -0.013311 0.676826 -1.218664 421.277039 1.000000 8903.778320 0.734644 0.313449 1.612776 00:25
16 3961.631104 4279.156250 3818.602783 -0.023979 0.654508 -1.387810 460.055573 1.000000 8851.189453 0.717633 0.323141 1.834598 00:25
17 3883.475586 4122.465820 3724.420898 -0.008015 0.687322 -1.149417 397.567566 1.000000 8689.211914 0.723197 0.310432 1.518600 00:25
18 3840.566650 4119.035156 3691.478027 -0.026009 0.669685 -1.277979 427.093781 1.000000 8522.090820 0.709724 0.320246 1.689855 00:25
19 3812.471191 4134.383301 3698.563232 -0.014498 0.668147 -1.309822 435.315338 1.000000 8758.106445 0.705087 0.324950 1.747292 00:25
20 3767.977051 3992.742676 3547.468750 -0.034419 0.659538 -1.350781 444.878937 1.000000 8189.853516 0.704693 0.324789 1.773977 00:25
21 3723.281494 3939.358887 3537.192627 -0.013502 0.689966 -1.189019 401.773529 1.000000 8170.947266 0.695311 0.325168 1.619005 00:25
22 3688.287109 4007.643555 3580.424805 -0.018447 0.674222 -1.281021 426.764069 1.000000 8233.244141 0.699469 0.327571 1.716607 00:26
23 3678.601562 3923.854248 3505.163086 -0.016415 0.674986 -1.260992 418.299408 1.000000 8065.615723 0.694784 0.326394 1.671506 00:25
24 3630.250244 3908.575195 3461.221436 -0.023672 0.666459 -1.361809 446.985535 1.000000 7965.235840 0.690277 0.335831 1.830896 00:25
25 3626.498291 3903.759521 3496.674316 -0.013907 0.686217 -1.209509 406.692993 1.000000 8029.478516 0.693128 0.329922 1.624238 00:25
26 3597.376465 3885.768066 3456.359863 -0.008869 0.673719 -1.306522 428.977020 1.000000 7959.312012 0.683002 0.335279 1.756485 00:25
27 3593.810059 3988.692871 3541.672119 -0.011080 0.669500 -1.339139 446.569672 1.000000 8099.977051 0.701862 0.337833 1.797264 00:25
28 3573.325195 3829.908447 3395.960693 -0.011860 0.675189 -1.295936 433.591003 1.000000 7721.243164 0.699376 0.335587 1.742384 00:26
29 3564.660889 3786.928711 3345.248535 -0.024480 0.668474 -1.339564 441.403625 1.000000 7804.659668 0.690355 0.338523 1.781669 00:25
30 3538.487793 3791.620361 3366.694336 -0.016601 0.676776 -1.271384 424.608124 1.000000 7792.783691 0.696108 0.333885 1.694794 00:25
31 3548.714844 3835.811523 3416.373779 -0.026706 0.682827 -1.228750 419.062561 1.000000 7792.014648 0.708988 0.331578 1.643911 00:25
32 3513.774902 3761.795898 3319.690430 -0.018128 0.669407 -1.317629 441.705902 1.000000 7464.705566 0.707018 0.336299 1.741212 00:25
33 3502.417969 3753.980469 3323.896729 -0.011950 0.676053 -1.294433 429.727722 1.000000 7533.951172 0.689708 0.338480 1.737193 00:26
34 3477.667969 3709.925049 3269.001709 -0.004544 0.668855 -1.335782 440.547546 1.000000 7495.550781 0.690255 0.340251 1.772696 00:25
35 3460.046631 3735.198975 3312.612793 -0.010980 0.679006 -1.271484 422.251831 1.000000 7543.686035 0.685776 0.339557 1.700740 00:25
36 3457.488770 3756.474609 3301.977539 -0.015253 0.663251 -1.366208 454.068909 1.000000 7613.058105 0.702964 0.341805 1.795775 00:25
37 3431.430176 3697.653564 3243.665283 -0.014459 0.669075 -1.351062 453.609192 1.000000 7452.788086 0.705302 0.345873 1.798131 00:25
38 3417.913086 3756.547607 3317.378418 -0.013135 0.669685 -1.324888 438.777130 1.000000 7666.058594 0.692281 0.342241 1.745103 00:25
39 3409.144043 3647.353027 3204.307373 -0.020123 0.668135 -1.331397 442.705261 1.000000 7376.371582 0.697968 0.341519 1.750849 00:25
40 3407.758301 3649.263428 3219.372070 -0.006288 0.682669 -1.265153 429.512146 1.000000 7415.768066 0.700749 0.343829 1.706248 00:25
41 3382.909668 3686.329590 3223.107910 -0.011264 0.663852 -1.383487 462.748718 1.000000 7278.731445 0.707191 0.346626 1.832180 00:25
42 3376.441406 3638.436035 3196.129883 -0.002653 0.672465 -1.340442 441.959717 1.000000 7526.583008 0.681990 0.349581 1.793306 00:25
43 3373.004639 3626.475586 3170.622314 -0.014423 0.665697 -1.368459 455.530701 1.000000 7291.094727 0.698909 0.349142 1.801970 00:25
44 3366.769287 3631.925293 3173.363770 -0.018959 0.659315 -1.397636 458.319580 1.000000 7270.541016 0.691233 0.347236 1.822071 00:25
45 3347.360840 3574.262451 3142.636963 -0.013239 0.668440 -1.326434 431.249786 1.000000 7364.531738 0.674849 0.342823 1.737871 00:25
46 3317.150879 3598.666748 3156.432617 -0.001533 0.668835 -1.350455 441.818359 1.000000 7342.520020 0.678997 0.347351 1.794104 00:25
47 3323.658203 3565.412109 3115.931641 -0.008903 0.667702 -1.356261 449.135681 1.000000 7219.029785 0.692640 0.346469 1.800012 00:25
48 3311.372559 3567.005615 3113.006104 -0.010357 0.668052 -1.343055 453.667999 1.000000 7082.435059 0.711734 0.346697 1.768867 00:25
49 3321.950195 3582.604004 3113.802490 0.000333 0.656901 -1.400587 468.360321 1.000000 7102.209473 0.714338 0.346219 1.814144 00:26
50 3301.511719 3623.174561 3182.668213 0.008003 0.671113 -1.321199 440.130829 1.000000 7262.528320 0.694835 0.346230 1.740687 00:25
51 3285.044678 3552.255371 3106.276123 -0.006173 0.669649 -1.354133 445.727448 1.000000 7177.645996 0.682762 0.350433 1.800857 00:25
52 3286.866455 3675.446045 3236.316650 -0.013157 0.676845 -1.295927 438.723022 1.000000 7318.729980 0.703599 0.346884 1.728112 00:25
53 3291.747803 3667.880127 3209.201904 -0.017041 0.670350 -1.326245 458.201050 1.000000 7180.023926 0.729949 0.348613 1.740342 00:25
54 3284.672852 3593.201660 3139.171631 -0.014269 0.670780 -1.354988 453.624573 1.000000 7211.184570 0.697685 0.352816 1.806872 00:25
55 3267.846924 3536.163086 3085.779541 -0.002470 0.663568 -1.371893 450.007965 1.000000 7252.326660 0.686323 0.348915 1.796228 00:25
56 3250.682617 3564.268555 3101.536377 -0.012916 0.666759 -1.372767 462.398376 1.000000 7097.330566 0.708008 0.353114 1.815818 00:25
57 3254.443604 3547.814453 3098.741943 -0.006862 0.665436 -1.359174 448.660187 1.000000 7157.088379 0.691026 0.348435 1.784791 00:26
58 3260.725586 3530.732178 3079.662109 -0.003880 0.665441 -1.359928 450.669525 1.000000 7183.651855 0.694633 0.348688 1.782578 00:25
59 3236.969482 3484.071289 3034.793945 0.002332 0.670223 -1.346266 448.978607 1.000000 7092.309082 0.694828 0.351385 1.783404 00:25
60 3225.157715 3552.310791 3117.349854 -0.007228 0.672280 -1.309015 434.601807 1.000000 7282.853027 0.690033 0.345595 1.724956 00:25
61 3220.063232 3524.829834 3085.624756 -0.005145 0.671900 -1.325763 438.820679 1.000000 7248.909180 0.686005 0.349257 1.752840 00:26
62 3227.847168 3484.733643 3023.396729 -0.007903 0.657956 -1.388762 461.027283 1.000000 6973.873535 0.704548 0.348345 1.789521 00:26
63 3190.125000 3485.780029 3038.699463 -0.006577 0.668708 -1.349016 446.739380 1.000000 7149.849121 0.689440 0.351118 1.780027 00:25
64 3197.777832 3448.150146 2999.025879 -0.004879 0.659723 -1.374128 448.797913 1.000000 6996.356934 0.686545 0.346901 1.771503 00:25
65 3207.785889 3439.828369 2991.286133 0.004332 0.666682 -1.338493 448.244476 1.000000 6964.046387 0.704034 0.347361 1.744932 00:25
66 3216.496582 3504.475586 3048.498779 0.005608 0.656708 -1.394906 455.563385 1.000000 7071.123535 0.688872 0.348882 1.791947 00:25
67 3205.509766 3455.002930 3015.201904 -0.002899 0.671343 -1.323578 439.482635 1.000000 7103.756348 0.689148 0.350410 1.740655 00:25
68 3190.522461 3437.476318 2982.573242 -0.001924 0.664799 -1.361731 454.539734 1.000000 6959.311523 0.702032 0.350246 1.781555 00:25
69 3179.375000 3434.706055 2972.019043 -0.000864 0.661195 -1.389103 462.347748 1.000000 6931.499023 0.702403 0.351853 1.810729 00:25
70 3154.015137 3450.184814 3001.612061 -0.000603 0.654439 -1.383468 448.301758 1.000000 7242.174316 0.685155 0.344582 1.757106 00:26
71 3109.754395 3367.349121 2874.592285 0.014566 0.634897 -1.472259 492.536652 1.000000 6862.613281 0.737525 0.344089 1.801952 00:26
72 3043.930176 3312.779297 2833.916260 -0.007480 0.644100 -1.434898 478.643097 1.000000 6766.743652 0.723580 0.348020 1.781140 00:25
73 2990.204834 3322.297363 2830.795410 -0.008267 0.642356 -1.456952 491.218353 1.000000 6806.497559 0.736002 0.350011 1.814443 00:25
74 2982.461426 3225.157471 2743.953613 -0.002834 0.637180 -1.477048 481.043793 1.000000 6698.075684 0.704703 0.349975 1.816552 00:25
75 2936.034180 3200.522705 2732.647705 -0.000358 0.648334 -1.434253 467.748627 1.000000 6595.842285 0.693321 0.353818 1.802200 00:25
76 2917.427979 3188.411133 2711.284668 -0.010846 0.649576 -1.430211 476.969299 1.000000 6641.311035 0.715748 0.352910 1.806378 00:26
77 2900.190918 3208.352539 2738.323975 0.001015 0.643907 -1.449119 469.884857 1.000000 6799.942383 0.692973 0.351222 1.807197 00:25
78 2893.122559 3170.847412 2691.765869 -0.006294 0.643069 -1.466224 479.063080 1.000000 6557.207520 0.699490 0.355875 1.826500 00:25
79 2879.305420 3165.599121 2672.646484 -0.009947 0.638078 -1.504574 492.835205 1.000000 6531.899414 0.706808 0.357110 1.871682 00:26
80 2871.717529 3141.771240 2661.048340 0.008516 0.644238 -1.472341 480.623749 1.000000 6581.002441 0.697051 0.357159 1.850578 00:26
81 2841.466553 3114.589111 2620.065918 -0.013839 0.643850 -1.486288 494.445160 1.000000 6409.358398 0.716239 0.360313 1.870289 00:26
82 2827.166260 3109.616699 2619.346680 -0.004788 0.640471 -1.500353 490.193634 1.000000 6416.575684 0.700102 0.360305 1.870650 00:25
83 2806.600342 3085.924561 2605.619385 0.007370 0.646208 -1.465798 480.237457 1.000000 6450.090820 0.697967 0.358896 1.845136 00:26
84 2791.562256 3063.895020 2580.600586 -0.005374 0.645183 -1.463034 483.229340 1.000000 6309.436035 0.707763 0.358645 1.829669 00:25
85 2770.278076 3034.631836 2558.484863 -0.010086 0.647094 -1.460025 476.043915 1.000000 6346.354004 0.691674 0.359322 1.838968 00:25
86 2755.365479 3014.786133 2529.739014 -0.004383 0.644172 -1.489430 484.962555 1.000000 6343.630371 0.692092 0.361628 1.875817 00:26
87 2738.068115 3026.836182 2534.303467 -0.006969 0.645632 -1.497038 492.442505 1.000000 6328.986816 0.700723 0.364204 1.900079 00:25
88 2720.661865 2986.312256 2502.743164 -0.004747 0.646495 -1.490602 483.493469 1.000000 6292.840332 0.683937 0.365157 1.889861 00:25
89 2698.882568 2965.148682 2481.424072 -0.002050 0.646837 -1.491359 483.654999 1.000000 6172.981934 0.683138 0.365614 1.893745 00:25
90 2688.093262 2944.035889 2457.755615 -0.002745 0.643649 -1.513131 486.249237 1.000000 6184.668945 0.675225 0.367282 1.911071 00:25
91 2674.913086 2943.592041 2454.669922 -0.002578 0.644348 -1.508578 488.867065 1.000000 6166.262207 0.683654 0.367843 1.904162 00:25
92 2655.663086 2926.737305 2436.727295 -0.002487 0.646099 -1.506547 489.977570 1.000000 6111.576660 0.685432 0.368832 1.912587 00:26
93 2640.149414 2910.103271 2416.333984 -0.002656 0.642178 -1.537301 493.722076 1.000000 6126.336426 0.674277 0.370405 1.945017 00:25
94 2629.382812 2903.057617 2412.663330 0.002669 0.642730 -1.537003 490.318329 1.000000 6109.676270 0.665981 0.370631 1.949367 00:25
95 2615.815918 2896.590576 2403.968506 -0.000958 0.644573 -1.534148 492.563263 1.000000 6112.826172 0.670643 0.372265 1.953958 00:26
96 2607.981445 2885.615967 2391.821045 -0.003212 0.642997 -1.540947 493.727631 1.000000 6075.826660 0.669907 0.372110 1.956079 00:25
97 2609.968506 2881.287109 2386.099365 -0.002940 0.642660 -1.548294 495.115814 1.000000 6065.745605 0.667622 0.372756 1.968524 00:25
98 2603.308350 2880.835693 2387.826660 -0.004268 0.643977 -1.538081 492.949677 1.000000 6068.658203 0.669138 0.372360 1.957721 00:25
99 2598.858887 2880.998047 2387.606201 -0.004501 0.643797 -1.540458 493.339844 1.000000 6068.282227 0.668343 0.372554 1.961107 00:25
{% endraw %} {% raw %}
prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')
{% endraw %}

64 latents, alpha=10

{% raw %}
latent_dim = 64
alpha = 10 # doubled because latent is half?
batchmean = True
useL1 = False


# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(0.00043651582673192023,
 0.00363078061491251,
 0.002033648220822215,
 0.0012589251855388284)
{% endraw %} {% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )
<fastai.learner.Learner at 0x7f2e8f2327f0>
{% endraw %} {% raw %}
# the defaults are pretty good for now
n_epochs = 10

learn.fit_one_cycle(n_epochs)#,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 58740.539062 55176.195312 54889.167969 0.120437 0.523871 -1.432346 5821.192383 0.047932 60959.058594 2.827253 0.179028 0.830196 00:25
1 45920.312500 32378.673828 32121.812500 -0.037095 0.680219 -0.876800 1361.199707 0.185387 44998.371094 1.317362 0.198534 0.791697 00:25
2 28567.009766 18783.580078 18389.089844 -0.023480 0.572010 -1.268163 1013.169067 0.385156 32817.027344 0.967404 0.211317 0.809108 00:26
3 19321.156250 13977.159180 13471.999023 -0.019137 0.579537 -1.282815 827.127136 0.607671 27876.337891 0.787131 0.226196 0.940032 00:25
4 14469.906250 12062.826172 11523.424805 -0.031412 0.642168 -1.077186 664.454651 0.808860 25497.568359 0.700015 0.236848 0.963814 00:25
5 11711.579102 10398.442383 9849.642578 -0.026665 0.686315 -0.960859 576.726868 0.948876 23399.960938 0.636489 0.249260 1.035778 00:25
6 10033.545898 9363.882812 8784.699219 -0.025187 0.691321 -0.990074 577.364075 0.999986 21912.365234 0.602564 0.263317 1.168671 00:25
7 9010.625977 8700.817383 8128.997070 -0.021991 0.701658 -0.985699 570.353088 1.000000 21030.861328 0.579129 0.276201 1.233390 00:25
8 8411.709961 8434.400391 7861.457031 -0.025626 0.711409 -0.979375 571.588013 1.000000 20555.650391 0.569511 0.285582 1.291906 00:25
9 8147.944824 8266.574219 7697.082031 -0.025811 0.712803 -0.977301 568.101562 1.000000 20270.447266 0.564367 0.286426 1.297384 00:26
{% endraw %}

This initial "burning in" of the KLD regularization is very unstable...

{% raw %}
learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()
<fastai.learner.Learner at 0x7f2e8f2327f0>
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
(6.309573450380412e-08,
 1.9054607491852948e-06,
 9.842782418445494e-07,
 3.467365559117752e-07)
{% endraw %} {% raw %}
base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final=1000.)
learn.show_results()
epoch train_loss valid_loss l2_b_mean mu std logvar weighted_kld kl_weight l1_b_mean mu_sd std_sd logvar_sd time
0 7651.367676 7365.542969 6727.640625 -0.001503 0.707371 -0.946704 636.378296 1.000000 18033.966797 0.688343 0.267682 1.189982 00:28
1 6716.003906 6474.808594 5877.452637 -0.036604 0.720139 -0.926342 595.863586 1.000000 16393.494141 0.638700 0.274323 1.235185 00:28
2 5997.029297 5860.405273 5274.698242 -0.018035 0.736100 -0.887804 584.582947 1.000000 15014.263672 0.634403 0.281981 1.247762 00:28
3 5485.274414 5364.577148 4783.366699 -0.004125 0.744810 -0.855897 580.197937 1.000000 13838.269531 0.644898 0.280916 1.228757 00:28
4 5094.323730 5222.379883 4581.991699 0.001645 0.730681 -0.953955 639.444214 1.000000 13211.501953 0.650297 0.294958 1.385436 00:28
5 4805.841309 4832.098145 4243.259277 -0.007232 0.739415 -0.903473 588.059143 1.000000 12172.434570 0.618609 0.291089 1.310845 00:28
6 4583.566406 4695.562012 4069.434326 0.008616 0.726104 -0.973470 625.385925 1.000000 11474.740234 0.621857 0.297524 1.394230 00:28
7 4416.959961 4745.934570 4140.187012 0.001199 0.740462 -0.918071 604.966614 1.000000 11435.058594 0.623564 0.298082 1.354706 00:28
8 4314.148438 4493.190918 3870.912598 -0.005136 0.733318 -0.939494 621.700745 1.000000 10547.795898 0.636232 0.296641 1.357139 00:28
9 4209.107910 4406.982422 3759.320068 -0.001849 0.727758 -1.000237 647.177917 1.000000 10461.577148 0.620679 0.307983 1.468478 00:28
10 4109.696777 4476.614258 3839.443359 -0.009990 0.732419 -0.969451 636.434265 1.000000 10169.422852 0.627681 0.304734 1.423959 00:28
11 4020.404785 4275.003906 3613.874756 -0.010094 0.728018 -1.003290 660.770203 1.000000 9553.526367 0.634482 0.308647 1.477842 00:28
12 3952.284912 4187.662598 3551.754150 0.005761 0.734981 -0.966541 635.446228 1.000000 9338.792969 0.623738 0.309741 1.428058 00:28
13 3900.007324 4116.317871 3462.226562 -0.006851 0.731415 -0.980627 653.731018 1.000000 8986.081055 0.639830 0.308371 1.442627 00:28
14 3851.263916 4204.222656 3555.328857 -0.012392 0.733164 -0.978597 648.536926 1.000000 9004.118164 0.632044 0.310562 1.446320 00:28
15 3785.325684 4007.054443 3348.741943 -0.000152 0.731587 -1.004563 658.063110 1.000000 8655.161133 0.622582 0.315733 1.497758 00:28
16 3767.278564 4183.640625 3517.904785 -0.008906 0.731445 -1.011055 665.384583 1.000000 8616.531250 0.626313 0.316394 1.513834 00:27
17 3758.536621 4103.028320 3443.043213 0.004418 0.725755 -1.007025 659.650574 1.000000 8621.183594 0.632305 0.310023 1.463498 00:28
18 3725.813477 3991.390381 3325.551514 -0.010483 0.731626 -1.013289 665.482483 1.000000 8288.931641 0.623968 0.317578 1.520346 00:28
19 3693.500732 3983.870850 3324.956543 0.013614 0.727927 -1.009194 658.678589 1.000000 8113.777832 0.625404 0.312643 1.483721 00:28
20 3654.010498 3962.879150 3288.293945 0.001136 0.722652 -1.044258 674.351501 1.000000 8213.806641 0.621986 0.315267 1.532625 00:28
21 3647.004395 3983.133789 3338.108398 0.004072 0.739812 -0.957298 644.710266 1.000000 8219.560547 0.635629 0.312068 1.439436 00:28
22 3617.129395 3887.343506 3229.427979 0.008313 0.730873 -1.007894 657.713196 1.000000 7895.292480 0.620299 0.315637 1.502362 00:28
23 3608.985107 3904.537842 3230.343018 0.000425 0.725290 -1.030399 674.001465 1.000000 7845.946289 0.629264 0.315511 1.516837 00:28
24 3591.437744 3859.909424 3193.618164 0.010155 0.734639 -1.009018 666.101379 1.000000 7719.414551 0.623905 0.318807 1.533824 00:28
25 3577.002197 3895.349121 3224.742920 -0.000528 0.729889 -1.021132 670.441528 1.000000 7743.109375 0.625357 0.318466 1.525949 00:28
26 3571.908447 3821.641602 3162.054688 0.000455 0.734451 -1.004335 659.434448 1.000000 7697.803711 0.619632 0.318668 1.517508 00:28
27 3548.191895 3849.494141 3180.026367 -0.004974 0.725175 -1.028136 669.249573 1.000000 7672.394043 0.626161 0.313801 1.516769 00:28
28 3533.383545 4059.682129 3408.733398 -0.000392 0.734781 -0.987317 650.586487 1.000000 8089.685059 0.623778 0.314767 1.479183 00:28
29 3516.743164 3760.632324 3109.681396 0.006230 0.734859 -0.989964 650.809082 1.000000 7584.377441 0.620539 0.317219 1.483018 00:28
30 3506.385498 3800.312500 3123.494629 -0.005099 0.726935 -1.031704 676.580688 1.000000 7519.890625 0.627771 0.318481 1.531833 00:28
31 3484.509277 3836.599365 3150.462402 -0.003075 0.719542 -1.075821 686.004456 1.000000 7661.994141 0.610878 0.322514 1.581861 00:28
32 3495.000732 3849.746582 3175.043701 0.010477 0.730324 -1.023581 674.504395 1.000000 7673.555176 0.626465 0.321434 1.533255 00:28
33 3487.385498 3826.780029 3129.293945 0.018080 0.733950 -1.026333 697.271606 1.000000 7434.991699 0.645661 0.325797 1.565082 00:28
34 3452.620361 3733.065918 3053.726318 0.004488 0.733624 -1.027590 679.199219 1.000000 7412.557617 0.622742 0.326465 1.560631 00:27
35 3455.627930 3729.029297 3067.841309 -0.000455 0.737738 -0.984265 660.988892 1.000000 7427.024414 0.632531 0.321030 1.482692 00:28
36 3448.309326 3716.426270 3055.599121 0.000706 0.732191 -1.013381 660.638794 1.000000 7478.499512 0.614729 0.322410 1.516803 00:28
37 3425.744873 3717.308594 3067.585449 0.009872 0.737455 -0.981283 649.612366 1.000000 7241.781738 0.621098 0.320518 1.472866 00:28
38 3406.716309 3655.440430 2981.873291 0.019556 0.730443 -1.028612 673.444092 1.000000 7264.535156 0.619335 0.324188 1.543146 00:28
39 3397.938721 3687.032959 2996.398438 0.017997 0.725269 -1.059344 690.533447 1.000000 7285.152832 0.619712 0.328644 1.572508 00:28
40 3396.500488 3704.924805 3025.917236 0.004099 0.733460 -1.027411 678.857178 1.000000 7252.134766 0.621408 0.328601 1.556395 00:28
41 3400.028320 3715.416748 3045.153564 0.004757 0.729243 -1.017588 670.118958 1.000000 7224.676270 0.626427 0.322481 1.500674 00:28
42 3389.938232 3709.358887 3035.591797 0.002872 0.730203 -1.026232 673.540894 1.000000 7237.694824 0.620630 0.326542 1.524723 00:28
43 3378.010010 3683.706787 2999.656250 0.010118 0.723173 -1.058739 683.975342 1.000000 7201.193359 0.614366 0.329194 1.549945 00:28
44 3378.157715 3711.697510 3017.652832 0.010882 0.726414 -1.050995 693.858459 1.000000 7120.746094 0.630194 0.327425 1.563918 00:27
45 3371.152588 3661.925781 2982.276611 0.002141 0.728925 -1.028453 679.527405 1.000000 7222.122070 0.628844 0.324787 1.527632 00:28
46 3362.568604 3678.501709 3016.866455 0.002026 0.734364 -1.003709 661.490845 1.000000 7162.108887 0.619610 0.324967 1.503220 00:28
47 3345.114502 3662.004883 3013.256836 0.007719 0.736944 -0.981593 648.579651 1.000000 7179.213867 0.619943 0.321520 1.468639 00:28
48 3339.086670 3722.939941 3025.527588 0.014281 0.727889 -1.059957 697.148499 1.000000 7148.197266 0.623049 0.331727 1.594466 00:28
49 3337.108643 3629.774658 2950.814941 0.008032 0.728444 -1.030823 678.777283 1.000000 7144.182617 0.625660 0.326432 1.524955 00:28
50 3311.899414 3642.291748 2961.773438 0.001066 0.730269 -1.038538 680.423950 1.000000 7062.062500 0.617276 0.330441 1.555513 00:28
51 3320.456543 3581.983154 2891.198242 0.006517 0.728271 -1.043807 690.645813 1.000000 6893.710449 0.627845 0.331142 1.549931 00:28
52 3307.604736 3677.112793 3016.974121 0.007375 0.736529 -0.997537 659.946533 1.000000 7134.888672 0.619345 0.326338 1.500275 00:28
53 3312.864990 3641.685059 2937.004639 -0.004953 0.720087 -1.083988 704.637512 1.000000 6979.736328 0.622097 0.331982 1.590897 00:28
54 3296.139648 3577.402588 2884.481689 0.004521 0.728107 -1.051127 692.779541 1.000000 7009.988770 0.624056 0.332423 1.568000 00:28
55 3282.488037 3654.056396 2947.325928 0.014008 0.723545 -1.081236 706.626526 1.000000 7055.623535 0.621703 0.333729 1.613418 00:28
56 3289.469238 3602.896240 2920.512939 0.006522 0.728315 -1.034476 682.367432 1.000000 6979.327637 0.626334 0.328607 1.531155 00:28
57 3292.510498 3604.759033 2907.411865 0.001527 0.724153 -1.068953 697.232239 1.000000 7029.240234 0.619716 0.333099 1.584040 00:28
58 3279.932373 3631.697266 2934.322021 0.013448 0.722786 -1.074533 697.274597 1.000000 6931.425781 0.616444 0.333294 1.585446 00:28
59 3263.800537 3766.281738 3084.604980 0.010203 0.727060 -1.042177 681.429016 1.000000 7250.323730 0.619173 0.330562 1.536164 00:28
60 3247.138428 3614.538818 2906.910400 0.012615 0.722017 -1.082184 707.588440 1.000000 6928.161621 0.622702 0.336169 1.592687 00:28
61 3261.955566 3540.338135 2835.608398 0.003212 0.724194 -1.073995 704.604065 1.000000 6790.403320 0.623425 0.335638 1.589854 00:27
62 3256.552002 3535.280029 2826.410645 0.001747 0.721265 -1.087617 708.741272 1.000000 6755.615723 0.621381 0.335210 1.605481 00:28
63 3250.792480 3548.330078 2842.145508 0.018147 0.726087 -1.072173 706.068848 1.000000 6790.458984 0.624347 0.335631 1.605922 00:28
64 3261.629150 3514.951660 2814.265869 0.005941 0.722353 -1.069727 700.618652 1.000000 6820.963379 0.625080 0.333472 1.567700 00:28
65 3241.566650 3608.032715 2904.027832 0.006998 0.723039 -1.076644 703.917480 1.000000 6986.800781 0.622058 0.335172 1.589766 00:28
66 3224.029053 3570.575195 2885.435303 0.016744 0.725186 -1.057620 685.048767 1.000000 6912.272949 0.612871 0.331627 1.565044 00:28
67 3251.290039 3583.719482 2884.775635 0.011005 0.723794 -1.060478 698.897400 1.000000 6890.466797 0.629185 0.332542 1.557823 00:28
68 3238.050049 3528.008789 2828.175049 0.019224 0.726878 -1.064695 699.730469 1.000000 6830.160645 0.621388 0.335719 1.589031 00:28
69 3223.922852 3530.111084 2814.860352 0.006388 0.723904 -1.079526 715.156250 1.000000 6741.392090 0.631759 0.336924 1.602475 00:28
70 3216.445068 3500.061523 2782.776855 0.008921 0.720420 -1.104010 717.167419 1.000000 6705.318359 0.617773 0.338750 1.635759 00:29
71 3207.145996 3521.456787 2817.231201 0.005418 0.720238 -1.086549 704.176575 1.000000 6791.708496 0.616982 0.336408 1.589192 00:28
72 3204.608154 3531.666260 2808.478760 0.009166 0.717051 -1.115761 723.127869 1.000000 6687.869629 0.620199 0.337613 1.643584 00:28
73 3213.811523 3598.167236 2889.819580 0.010586 0.726090 -1.083778 708.206543 1.000000 6976.821777 0.615494 0.339968 1.628908 00:28
74 3203.632324 3670.336182 2951.901855 0.005238 0.719610 -1.084969 718.238342 1.000000 6905.762695 0.637379 0.334413 1.585946 00:28
75 3207.873535 3579.372803 2876.421875 0.006835 0.723253 -1.067832 702.820068 1.000000 6790.896484 0.627843 0.334451 1.570162 00:28
76 3187.589355 3531.233643 2829.830811 0.013975 0.722816 -1.077619 701.346497 1.000000 6883.243164 0.618706 0.334157 1.596420 00:28
77 3181.636719 3545.495117 2852.156006 0.002698 0.731067 -1.049801 693.244995 1.000000 6794.460449 0.620472 0.335831 1.587278 00:28
78 3177.850098 3536.807861 2822.090088 0.005285 0.719301 -1.111130 714.644043 1.000000 6781.951172 0.608447 0.341650 1.637716 00:28
79 3180.280518 3572.795410 2873.986572 0.006143 0.722267 -1.085594 698.616516 1.000000 6843.494141 0.607717 0.337304 1.603625 00:28
80 3168.005371 3535.424561 2833.231201 0.019834 0.726783 -1.064308 702.152222 1.000000 6706.426758 0.623918 0.337304 1.585141 00:28
81 3152.529297 3446.023438 2734.945557 0.008887 0.723976 -1.088619 710.974243 1.000000 6700.422852 0.617348 0.340108 1.623608 00:28
82 3135.922119 3434.467773 2714.553223 0.012614 0.720932 -1.108265 719.850525 1.000000 6591.262695 0.614366 0.343730 1.639695 00:27
83 3113.646729 3466.451172 2765.960205 0.003383 0.726390 -1.076803 700.406921 1.000000 6657.192871 0.610935 0.339992 1.610990 00:28
84 3098.293701 3459.226318 2755.147705 0.016013 0.724201 -1.087780 703.987000 1.000000 6683.789551 0.607959 0.341545 1.616467 00:28
85 3087.032959 3389.620605 2680.396240 0.007750 0.721710 -1.108647 709.155762 1.000000 6485.849121 0.599022 0.344326 1.646617 00:28
86 3070.500977 3382.860840 2672.243164 0.014384 0.723655 -1.103893 710.518250 1.000000 6548.608398 0.602184 0.344518 1.650620 00:28
87 3055.568359 3381.640869 2666.888916 0.006845 0.725388 -1.102032 714.671509 1.000000 6474.822754 0.606631 0.345363 1.659149 00:28
88 3045.261719 3402.307373 2680.911377 0.006631 0.723334 -1.119401 721.316833 1.000000 6557.485352 0.602261 0.347152 1.684888 00:28
89 3029.100830 3327.909180 2613.521240 0.009250 0.723729 -1.116796 714.319580 1.000000 6478.024902 0.594214 0.348252 1.677276 00:28
90 3009.680908 3317.802979 2615.427002 0.011575 0.728394 -1.091922 702.291748 1.000000 6491.918945 0.594770 0.345983 1.656690 00:28
91 2996.286865 3309.975342 2594.326660 0.006130 0.726961 -1.113291 715.553467 1.000000 6424.102539 0.593813 0.349996 1.692982 00:28
92 2977.406250 3292.775391 2571.700195 0.013427 0.725092 -1.122516 721.000305 1.000000 6330.194824 0.594845 0.350828 1.698746 00:27
93 2972.783447 3268.155273 2544.673096 0.008637 0.725328 -1.129471 723.458557 1.000000 6335.770508 0.591184 0.352287 1.717149 00:28
94 2970.029785 3270.115967 2551.228760 0.009532 0.724969 -1.130088 718.871765 1.000000 6373.247559 0.585079 0.352135 1.716223 00:28
95 2954.962891 3261.302979 2542.979004 0.008221 0.725357 -1.124736 718.260071 1.000000 6304.031250 0.588493 0.351823 1.705143 00:28
96 2950.425049 3252.043213 2532.653076 0.010782 0.725520 -1.130057 719.378235 1.000000 6317.508301 0.584415 0.353175 1.717738 00:28
97 2943.955322 3251.385254 2532.198486 0.008705 0.726147 -1.125400 719.156494 1.000000 6292.142090 0.587820 0.352392 1.712947 00:28
98 2944.449463 3245.879883 2527.619141 0.008804 0.725314 -1.130287 718.228455 1.000000 6309.519043 0.583112 0.352947 1.717500 00:28
99 2937.538330 3244.074463 2529.292725 0.008952 0.725937 -1.125587 714.744995 1.000000 6312.990234 0.581985 0.352475 1.711296 00:28
{% endraw %} {% raw %}
prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')
{% endraw %}

CODE graveyard

{% raw %}
class LatentTuple(fastuple):
    "Basic type for tuple of tensor (vectors)"
    _show_args = dict(s=10, marker='.', c='r')
    @classmethod
    def create(cls, ts): 
        if isinstance(ts,tuple):
            mu,logvar = ts
        elif ts is None:
            mu,logvar = None,None
        else:
            mu = None
            logvar = None
            
        if mu is None: mu = torch.empty(0)
        elif not isinstance(mu, Tensor): Tensor(mu) 
        
        if logvar is None: logvar = torch.empty(0)
        elif not isinstance(logvar,Tensor): Tensor(logvar)
            
        return cls( (mu,logvar) ) 
        
    def show(self, ctx=None, **kwargs): 
        mu,logvar = self
        if not isinstance(mu, Tensor) or not isinstance(logvar,Tensor): return ctx

        title_str = f"mu-> {mu.mean():e}, {mu.std():e}  logvar->{logvar.mean():e}, {logvar.std():e}"
    
        if 'figsize' in kwargs: del kwargs['figsize']
        if 'title' in kwargs: kwargs['title']=title_str
        if ctx is None:
            _,axs = plt.subplots(1,2, figsize=(12,6))
            x=torch.linspace(0,1,mu[0].shape[0])
            axs[0].scatter(x, mu[:], **{**self._show_args, **kwargs})
            axs[1].scatter(x, logvar[:], **{**self._show_args, **kwargs})
            ctx = axs[1]
        
        ctx.scatter(mu[:], logvar[:], **{**self._show_args, **kwargs})
        return ctx
    
# could we do a typedispatch to manage the transforms...?
def VAETargetTupleBlock(): 
    return TransformBlock(type_tfms=VAETargetTuple.create, batch_tfms=IntToFloatTensor)

def LatentTupleBlock(): 
    return TransformBlock(type_tfms=LatentTuple.create, batch_tfms=noop)
    

# class TensorPoint(TensorBase):
#     "Basic type for points in an image"
#     _show_args = dict(s=10, marker='.', c='r')

#     @classmethod
#     def create(cls, t, img_size=None)->None:
#         "Convert an array or a list of points `t` to a `Tensor`"
#         return cls(tensor(t).view(-1, 2).float(), img_size=img_size)

#     def show(self, ctx=None, **kwargs):
#         if 'figsize' in kwargs: del kwargs['figsize']
#         x = self.view(-1,2)
#         ctx.scatter(x[:, 0], x[:, 1], **{**self._show_args, **kwargs})
#         return ctx
{% endraw %} {% raw %}
latent_dim = 128
dropout = .2
im_size = IMG_SIZE
n_blocks = 5        
nfs = [3] + [2**i*n_blocks for i in range(n_blocks+1)] 
nfs.reverse()
# decoder = nn.Sequential(
#             nn.Linear(latent_size, 16),
#             UnFlatten(4),
#             ResBlock(1, 3, 4, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, 4, 8, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, 8, 16, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             #nn.AdaptiveAvgPool2d((3,im_size, im_size)) 
# )





n_blocks = 5
hidden_dim = 2048
out_range = [-1,1]
tst = nn.Sequential(
            nn.Linear(latent_dim,hidden_dim), #nn.Linear(latent_dim, 16)
            nn.Linear(hidden_dim,im_size*n_blocks*n_blocks), #nn.Linear(latent_dim, 16)
            ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[4], nfs[5], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[5], nfs[6], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            #nn.Dropout2d(dropout),
            #nn.Upsample(scale_factor=2),            #
            #nn.AdaptiveAvgPool2d((3,im_size, im_size)),
            SigmoidRange(*out_range), #nn.Sigmoid()
)
tst,nfs
{% endraw %} {% raw %}
inp=    torch.randn((32,latent_dim))
#last_size = model_sizes(tst ) #[-1][1]
#num_features_model(tst)
#last_size
#nfs
tst(inp).shape,nfs
#last_size
{% endraw %} {% raw %}
z_dim = 100
enc = nn.Sequential(
    ResBlock(1, 1, 16, act_cls=nn.ReLU, norm_type=None),
    nn.MaxPool2d(2, 2),
    ResBlock(1, 16, 4, act_cls=nn.ReLU, norm_type=None),
    nn.MaxPool2d(2, 2),
    Flatten()
)
# torch.Size([32, 1, 28, 28])
# torch.Size([32, 16, 28, 28])
# torch.Size([32, 16, 14, 14])
# torch.Size([32, 4, 14, 14])
# torch.Size([32, 4, 7, 7])
# torch.Size([32, 196])

latent_size = 100

enc = nn.Sequential(
            ResBlock(1, 3, 5, stride=2, act_cls=Mish),#  1->3
            ResBlock(1, 5, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 1, stride=2, act_cls=Mish),
            Flatten(),
            nn.Linear(400, latent_size) # 16->400
        )
#  torch.Size([32, 1, 28, 28])
# torch.Size([32, 5, 14, 14])
# torch.Size([32, 5, 7, 7])
# torch.Size([32, 1, 4, 4])
# torch.Size([32, 16])
# torch.Size([32, 4])       
inp=    torch.randn((32,3,160,160))
for ii in range(0,8):
    print(enc[:ii](inp).shape)

z = enc(inp)
{% endraw %} {% raw %}
dropout=0
dec = nn.Sequential(
            nn.Linear(latent_size, 16),
            UnFlatten(4),
            ResBlock(1, 1, 4, act_cls=Mish), #4->5
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 4, 8, act_cls=Mish),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 8, 16, act_cls=Mish),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(16, 3, 3), #1->3
            #nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((28, 28)),
            nn.Sigmoid()
        )

# torch.Size([32, 4])
# torch.Size([32, 16])
# torch.Size([32, 1, 4, 4])
# torch.Size([32, 4, 4, 4])
# torch.Size([32, 4, 8, 8])
# torch.Size([32, 8, 8, 8])
# torch.Size([32, 8, 16, 16])
# torch.Size([32, 16, 16, 16])
# torch.Size([32, 16, 32, 32])
# torch.Size([32, 1, 30, 30])
# torch.Size([32, 1, 28, 28])


for ii in range(0,12):
    print(dec[:ii](z).shape)
{% endraw %} {% raw %}
n_blocks = 5
BASE = im_size//2**5        
nfs = [3]+[(2**i)*BASE for i in range(n_blocks)] 
n = len(nfs)
hidden_dim = 2048

BASE = im_size//2**5
        # encoder
in_dim = nfs[-1] * BASE**2 
        
modules =  [ResBlock(1, nfs[i],nfs[i+1], 
                          stride=2, act_cls=Mish)  for i in range(n - 1)]    
# enc =  nn.Sequential(
#                 ConvLayer(nfs[0],nfs[1],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[1],nfs[2],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[2],nfs[3],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[3],nfs[4],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[4],nfs[5],ks=5,stride=2,padding=2),
#                 Flatten(),
#                 LinBnDrop(in_dim,hidden_dim,bn=True,p=0.0,act=nn.ReLU(),lin_first=True)
#             )

enc =  nn.Sequential(*modules,
                Flatten(),
                LinBnDrop(in_dim,hidden_dim,bn=True,p=0.0,act=nn.ReLU(),lin_first=True)
            )

nfs.reverse()

print(nfs)
#last_size = model_sizes(enc, size=(28,28))[-1][1]



    
encoder = nn.Sequential(enc, nn.Linear(hidden_dim, z_dim))
        
decoder = nn.Sequential(
            nn.Linear(z_dim, im_size*n_blocks*n_blocks),
            ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], ks=1, act_cls=nn.ReLU, norm_type=None),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=nn.ReLU, norm_type=None),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(nfs[2], 3, 3, padding=1),
            #nn.Dropout2d(dropout),
            nn.Sigmoid()
        )
#last_size

#             nn.Linear(latent_dim,hidden_dim), #nn.Linear(latent_dim, 16)
#             nn.Linear(hidden_dim,im_size*n_blocks*n_blocks), #nn.Linear(latent_dim, 16)
#             ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
#             ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[4], nfs[5], act_cls=Mish), #nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[5], nfs[6], act_cls=Mish), #nn.Conv2d(16, 1, 3),

            

inp=    torch.randn((32,3,160,160))

#encoder[:1](inp).shape
for ii in range(0,10):
    print(enc[:ii](inp).shape)

z = encoder(inp)
for ii in range(0,14):
    print(decoder[:ii](z).shape)
{% endraw %} {% raw %}
class UnFlatten(Module):
    def __init__(self, size=7):
        self.size = size
    def forward(self, input):
        return input.view(input.size(0), -1, self.size, self.size)

class MMD_VAE(Module):
    def __init__(self, latent_size):        
        self.encoder = nn.Sequential(
            ResBlock(1, 1, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 1, stride=2, act_cls=Mish),
            Flatten(),
            nn.Linear(16, latent_size)
        )
        
        dropout=0
        
        self.decoder = nn.Sequential(
            nn.Linear(latent_size, 16),
            UnFlatten(4),
            ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((28, 28)),
            nn.Sigmoid()
        )
        
    
    def forward(self, X):
        latent = self.encoder(X)
        return self.decoder(latent), latent
{% endraw %} {% raw %}
    
#decoder
n_blocks = 5        
nfs = [3] + [2**i*n_blocks for i in range(n_blocks+1)] 
nfs.reverse()
n = len(nfs)


tst = nn.Sequential(
            nn.Linear(latent_dim,hidden_dim, #nn.Linear(latent_dim, 16)
            nn.Linear(hidden_dim,im_size*n_blocks*n_blocks) #nn.Linear(latent_dim, 16)
            UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((3,im_size, im_size)),
            SigmoidRange(*out_range)#nn.Sigmoid()

    
                                         *modules,
                                      ConvLayer(nfs[-2],nfs[-1],
                                                ks=1,padding=0, norm_type=None, #act_cls=nn.Sigmoid) )
                                                act_cls=partial(SigmoidRange, *out_range)))
    
{% endraw %} {% raw %}
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr
{% endraw %} {% raw %}
n_epoch = 10
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()
{% endraw %} {% raw %}
n_epoch = 40
#learn.unfreeze()
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.05)

#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()
{% endraw %} {% raw %}
prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"frozen{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')
{% endraw %} {% raw %}
 
{% endraw %} {% raw %}
x=1
x
{% endraw %} {% raw %}
def create_encoder(nfs,ks,conv=nn.Conv2d,bn=nn.BatchNorm2d,act_fn = nn.ReLU):
    """
    constructor for generic convolutional encoder 
    """
    n = len(nfs)
    conv_layers = [nn.Sequential(ConvBnRelu(nfs[i],nfs[i+1],kernel_size=ks[i],
                                            conv = conv,bn=bn,act_fn=act_fn, padding = ks[i] //2 ),
                                 Downsample(channels=nfs[i+1],filt_size=3,stride=2))
                                   for i in range(n-1)]        
    convs = nn.Sequential(*conv_layers)
    return convs

def create_encoder_denseblock(n_dense,c_start):
    """
    constructor for resnet with dense blocks  (?) 

    n_dense": 3,
    "c_start": 4
    """
    first_layer = nn.Sequential(ConvBnRelu(3,c_start,kernel_size=3,padding = 1),
                                ResBlock(c_start),
                                Downsample(channels=4,filt_size=3,stride=2))
    
    layers = [first_layer] + [
        nn.Sequential(
            DenseBlock(c_start * (2**c)),
            Downsample(channels=c_start * (2**(c+1)),filt_size=3,stride=2)) for c in range(n_dense)
    ]
    
    model = nn.Sequential(*layers)
    
    return model

def create_decoder(nfs, ks, size, conv=nn.Conv2d, bn=nn.BatchNorm2d, act_fn=nn.ReLU):
    """
    CURR VALUES:
    "nfs":[66,3*32,3*16,3*8,3*4,3*2,3,1,3],
    "ks": [ 3, 1, 3,1,3,1,3,1],   
    "size": IMG_SIZE 
    """
    n = len(nfs)
    
    # We add two channels to the first layer to include x and y channels
    first_layer = ConvBnRelu(nfs[0], #input size 
                             nfs[1], # output size
                             conv = PointwiseConv,
                             bn=bn,
                             act_fn=act_fn)

    conv_layers = [first_layer] + [ConvBnRelu(nfs[i],nfs[i+1],kernel_size=ks[i-1],
                                              padding = ks[i-1] // 2,conv = conv,bn=bn,act_fn=act_fn)
                                   for i in range(1,n - 1)]        
    dec_convs = nn.Sequential(*conv_layers)
    
    dec = nn.Sequential(SpatialDecoder2D(size),dec_convs)
    #SigmoidRange(*y_range)
    return dec

def decoder_simple(y_range=OUT_RANGE, n_out=3):
    return nn.Sequential(#UpsampleBlock(64),
                         UpsampleBlock(32),
                         nn.Conv2d(16, n_out, 1),
                         SigmoidRange(*y_range)
                        )
{% endraw %}